Login
[x]
Log in using an account from:
Fedora Account System
Red Hat Associate
Red Hat Customer
Or login using a Red Hat Bugzilla account
Forgot Password
Login:
Hide Forgot
Create an Account
Red Hat Bugzilla – Attachment 936121 Details for
Bug 1140175
unable to create rhevm bridge through jsonrpc
[?]
New
Simple Search
Advanced Search
My Links
Browse
Requests
Reports
Current State
Search
Tabular reports
Graphical reports
Duplicates
Other Reports
User Changes
Plotly Reports
Bug Status
Bug Severity
Non-Defaults
|
Product Dashboard
Help
Page Help!
Bug Writing Guidelines
What's new
Browser Support Policy
5.0.4.rh83 Release notes
FAQ
Guides index
User guide
Web Services
Contact
Legal
This site requires JavaScript to be enabled to function correctly, please enable it.
engine.log
engine.log (text/plain), 2.56 MB, created by
Petr Beňas
on 2014-09-10 12:21:18 UTC
(
hide
)
Description:
engine.log
Filename:
MIME Type:
Creator:
Petr Beňas
Created:
2014-09-10 12:21:18 UTC
Size:
2.56 MB
patch
obsolete
>2014-09-09 15:24:44,678 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Loaded file "/usr/share/ovirt-engine/services/ovirt-engine/ovirt-engine.conf". >2014-09-09 15:24:44,678 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) The file "/etc/ovirt-engine/engine.conf" doesn't exist or isn't readable. Will return an empty set of properties. >2014-09-09 15:24:44,679 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Loaded file "/etc/ovirt-engine/engine.conf.d/10-rhevm-doc.conf". >2014-09-09 15:24:44,691 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Loaded file "/etc/ovirt-engine/engine.conf.d/10-setup-database.conf". >2014-09-09 15:24:44,691 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Loaded file "/etc/ovirt-engine/engine.conf.d/10-setup-jboss.conf". >2014-09-09 15:24:44,691 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Loaded file "/etc/ovirt-engine/engine.conf.d/10-setup-pki.conf". >2014-09-09 15:24:44,692 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Loaded file "/etc/ovirt-engine/engine.conf.d/10-setup-protocols.conf". >2014-09-09 15:24:44,692 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Loaded file "/etc/ovirt-engine/engine.conf.d/20-branding-rhev.conf". >2014-09-09 15:24:44,692 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Loaded file "/etc/ovirt-engine/engine.conf.d/50-redhat-support-plugin.conf". >2014-09-09 15:24:44,693 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Value of property "ENGINE_AJP_ENABLED" is "true". >2014-09-09 15:24:44,693 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Value of property "ENGINE_AJP_PORT" is "8702". >2014-09-09 15:24:44,695 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Value of property "ENGINE_APPS" is "engine.ear restapi.war legacy_restapi.war "/usr/share/ovirt-engine/branding/rhev-2.brand/applications/rhev.ear" redhat-support-plugin-rhev.ear". >2014-09-09 15:24:44,696 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Value of property "ENGINE_CACHE" is "/var/cache/ovirt-engine". >2014-09-09 15:24:44,697 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Value of property "ENGINE_DB_CHECK_INTERVAL" is "1000". >2014-09-09 15:24:44,697 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Value of property "ENGINE_DB_CONNECTION_TIMEOUT" is "300000". >2014-09-09 15:24:44,697 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Value of property "ENGINE_DB_DATABASE" is "engine". >2014-09-09 15:24:44,704 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Value of property "ENGINE_DB_DRIVER" is "org.postgresql.Driver". >2014-09-09 15:24:44,705 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Value of property "ENGINE_DB_HOST" is "localhost". >2014-09-09 15:24:44,705 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Value of property "ENGINE_DB_MAX_CONNECTIONS" is "100". >2014-09-09 15:24:44,705 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Value of property "ENGINE_DB_MIN_CONNECTIONS" is "1". >2014-09-09 15:24:44,705 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Value of property "ENGINE_DB_PASSWORD" is "***". >2014-09-09 15:24:44,705 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Value of property "ENGINE_DB_PORT" is "5432". >2014-09-09 15:24:44,705 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Value of property "ENGINE_DB_SECURED" is "False". >2014-09-09 15:24:44,706 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Value of property "ENGINE_DB_SECURED_VALIDATION" is "False". >2014-09-09 15:24:44,706 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Value of property "ENGINE_DB_URL" is "jdbc:postgresql://localhost:5432/engine?sslfactory=org.postgresql.ssl.NonValidatingFactory". >2014-09-09 15:24:44,706 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Value of property "ENGINE_DB_USER" is "engine". >2014-09-09 15:24:44,706 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Value of property "ENGINE_DEBUG_ADDRESS" is "". >2014-09-09 15:24:44,706 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Value of property "ENGINE_DOC" is "/usr/share/doc/ovirt-engine". >2014-09-09 15:24:44,707 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Value of property "ENGINE_ETC" is "/etc/ovirt-engine". >2014-09-09 15:24:44,707 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Value of property "ENGINE_EXTENSION_PATH" is "/usr/share/ovirt-engine/extensions.d:/etc/ovirt-engine/extensions.d". >2014-09-09 15:24:44,707 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Value of property "ENGINE_FQDN" is "pb-rh35.rhev.lab.eng.brq.redhat.com". >2014-09-09 15:24:44,707 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Value of property "ENGINE_GROUP" is "ovirt". >2014-09-09 15:24:44,707 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Value of property "ENGINE_HEAP_MAX" is "1g". >2014-09-09 15:24:44,707 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Value of property "ENGINE_HEAP_MIN" is "1g". >2014-09-09 15:24:44,708 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Value of property "ENGINE_HTTPS_ENABLED" is "false". >2014-09-09 15:24:44,708 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Value of property "ENGINE_HTTPS_PORT" is "None". >2014-09-09 15:24:44,708 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Value of property "ENGINE_HTTPS_PROTOCOLS" is "SSLv3,TLSv1,TLSv1.1,TLSv1.2". >2014-09-09 15:24:44,708 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Value of property "ENGINE_HTTP_ENABLED" is "false". >2014-09-09 15:24:44,714 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Value of property "ENGINE_HTTP_PORT" is "None". >2014-09-09 15:24:44,714 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Value of property "ENGINE_JAVA_MODULEPATH" is "/usr/share/ovirt-engine/modules". >2014-09-09 15:24:44,714 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Value of property "ENGINE_JVM_ARGS" is " -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath="/var/log/ovirt-engine/dump"". >2014-09-09 15:24:44,714 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Value of property "ENGINE_LOG" is "/var/log/ovirt-engine". >2014-09-09 15:24:44,714 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Value of property "ENGINE_LOG_TO_CONSOLE" is "false". >2014-09-09 15:24:44,715 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Value of property "ENGINE_MANUAL" is "/usr/share/doc/rhevm-doc/manual". >2014-09-09 15:24:44,716 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Value of property "ENGINE_PERM_MAX" is "256m". >2014-09-09 15:24:44,716 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Value of property "ENGINE_PERM_MIN" is "256m". >2014-09-09 15:24:44,716 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Value of property "ENGINE_PKI" is "/etc/pki/ovirt-engine". >2014-09-09 15:24:44,716 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Value of property "ENGINE_PKI_CA" is "/etc/pki/ovirt-engine/ca.pem". >2014-09-09 15:24:44,716 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Value of property "ENGINE_PKI_ENGINE_CERT" is "/etc/pki/ovirt-engine/certs/engine.cer". >2014-09-09 15:24:44,716 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Value of property "ENGINE_PKI_ENGINE_STORE" is "/etc/pki/ovirt-engine/keys/engine.p12". >2014-09-09 15:24:44,716 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Value of property "ENGINE_PKI_ENGINE_STORE_ALIAS" is "1". >2014-09-09 15:24:44,717 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Value of property "ENGINE_PKI_ENGINE_STORE_PASSWORD" is "***". >2014-09-09 15:24:44,717 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Value of property "ENGINE_PKI_TRUST_STORE" is "/etc/pki/ovirt-engine/.truststore". >2014-09-09 15:24:44,717 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Value of property "ENGINE_PKI_TRUST_STORE_PASSWORD" is "***". >2014-09-09 15:24:44,717 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Value of property "ENGINE_PROPERTIES" is " jsse.enableSNIExtension=false "java.security.krb5.conf=/etc/ovirt-engine/krb5.conf"". >2014-09-09 15:24:44,725 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Value of property "ENGINE_PROXY_ENABLED" is "true". >2014-09-09 15:24:44,725 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Value of property "ENGINE_PROXY_HTTPS_PORT" is "443". >2014-09-09 15:24:44,726 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Value of property "ENGINE_PROXY_HTTP_PORT" is "80". >2014-09-09 15:24:44,726 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Value of property "ENGINE_REPORTS_UI" is "/var/lib/ovirt-engine/reports.xml". >2014-09-09 15:24:44,726 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Value of property "ENGINE_STOP_INTERVAL" is "1". >2014-09-09 15:24:44,726 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Value of property "ENGINE_STOP_TIME" is "10". >2014-09-09 15:24:44,726 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Value of property "ENGINE_TMP" is "/var/tmp/ovirt-engine". >2014-09-09 15:24:44,726 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Value of property "ENGINE_UP_MARK" is "/var/lib/ovirt-engine/engine.up". >2014-09-09 15:24:44,727 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Value of property "ENGINE_URI" is "/ovirt-engine". >2014-09-09 15:24:44,727 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Value of property "ENGINE_USER" is "ovirt". >2014-09-09 15:24:44,727 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Value of property "ENGINE_USR" is "/usr/share/ovirt-engine". >2014-09-09 15:24:44,727 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Value of property "ENGINE_VAR" is "/var/lib/ovirt-engine". >2014-09-09 15:24:44,727 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Value of property "ENGINE_VERBOSE_GC" is "false". >2014-09-09 15:24:44,727 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Value of property "JBOSS_HOME" is "/usr/share/jbossas". >2014-09-09 15:24:44,727 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Value of property "JBOSS_RUNTIME" is "/var/lib/ovirt-engine/jboss_runtime". >2014-09-09 15:24:44,728 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Value of property "SENSITIVE_KEYS" is ",ENGINE_DB_PASSWORD,ENGINE_PKI_TRUST_STORE_PASSWORD,ENGINE_PKI_ENGINE_STORE_PASSWORD". >2014-09-09 15:25:11,957 INFO [org.ovirt.engine.core.bll.Backend] (ServerService Thread Pool -- 51) Start initializing Backend >2014-09-09 15:25:12,607 INFO [org.ovirt.engine.core.utils.osinfo.OsInfoPreferencesLoader] (ServerService Thread Pool -- 51) Loaded file /etc/ovirt-engine/osinfo.conf.d/00-defaults.properties >2014-09-09 15:25:12,795 INFO [org.ovirt.engine.core.bll.Backend] (ServerService Thread Pool -- 51) Running ovirt-engine 3.5.0-0.10.master.el6ev >2014-09-09 15:25:12,795 INFO [org.ovirt.engine.core.bll.CpuFlagsManagerHandler] (ServerService Thread Pool -- 51) Start initializing dictionaries >2014-09-09 15:25:12,804 INFO [org.ovirt.engine.core.bll.CpuFlagsManagerHandler] (ServerService Thread Pool -- 51) Finished initializing dictionaries >2014-09-09 15:25:12,804 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (ServerService Thread Pool -- 51) Start initializing AuditLogCleanupManager >2014-09-09 15:25:12,805 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (ServerService Thread Pool -- 51) Setting audit cleanup manager to run at: 35 35 3 * * ? >2014-09-09 15:25:12,832 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (ServerService Thread Pool -- 51) Finished initializing AuditLogCleanupManager >2014-09-09 15:25:12,833 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (ServerService Thread Pool -- 51) Start initializing CommandEntityCleanupManager >2014-09-09 15:25:12,833 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (ServerService Thread Pool -- 51) Setting command entity cleanup manager to run at: 35 35 3 * * ? >2014-09-09 15:25:12,840 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (ServerService Thread Pool -- 51) Finished initializing CommandEntityCleanupManager >2014-09-09 15:25:12,844 INFO [org.ovirt.engine.core.bll.TagsDirector] (ServerService Thread Pool -- 51) Start initializing TagsDirector >2014-09-09 15:25:12,859 INFO [org.ovirt.engine.core.bll.TagsDirector] (ServerService Thread Pool -- 51) Tag root added to tree >2014-09-09 15:25:12,877 INFO [org.ovirt.engine.core.bll.TagsDirector] (ServerService Thread Pool -- 51) Finished initializing TagsDirector >2014-09-09 15:25:12,877 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (ServerService Thread Pool -- 51) Start initializing IsoDomainListSyncronizer >2014-09-09 15:25:12,884 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (ServerService Thread Pool -- 51) Finished initializing IsoDomainListSyncronizer >2014-09-09 15:25:12,926 INFO [org.ovirt.engine.core.bll.Backend] (ServerService Thread Pool -- 51) Completed initializing handlers >2014-09-09 15:25:12,953 INFO [org.ovirt.engine.core.utils.ErrorTranslatorImpl] (ServerService Thread Pool -- 51) Start initializing ErrorTranslatorImpl >2014-09-09 15:25:12,967 WARN [org.ovirt.engine.core.utils.ErrorTranslatorImpl] (ServerService Thread Pool -- 51) Code MAC_ADDRESS_IS_IN_USE appears more than once in string table. >2014-09-09 15:25:12,968 INFO [org.ovirt.engine.core.utils.ErrorTranslatorImpl] (ServerService Thread Pool -- 51) Finished initializing ErrorTranslatorImpl >2014-09-09 15:25:12,968 INFO [org.ovirt.engine.core.utils.ErrorTranslatorImpl] (ServerService Thread Pool -- 51) Start initializing ErrorTranslatorImpl >2014-09-09 15:25:12,969 INFO [org.ovirt.engine.core.utils.ErrorTranslatorImpl] (ServerService Thread Pool -- 51) Finished initializing ErrorTranslatorImpl >2014-09-09 15:25:12,969 INFO [org.ovirt.engine.core.bll.Backend] (ServerService Thread Pool -- 51) Mark incomplete jobs as UNKNOWN >2014-09-09 15:25:13,020 INFO [org.ovirt.engine.core.bll.job.JobRepositoryCleanupManager] (ServerService Thread Pool -- 51) Start initializing JobRepositoryCleanupManager >2014-09-09 15:25:13,020 INFO [org.ovirt.engine.core.bll.job.JobRepositoryCleanupManager] (ServerService Thread Pool -- 51) Finished initializing JobRepositoryCleanupManager >2014-09-09 15:25:13,021 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (ServerService Thread Pool -- 51) Start initializing AutoRecoveryManager >2014-09-09 15:25:13,022 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (ServerService Thread Pool -- 51) Finished initializing AutoRecoveryManager >2014-09-09 15:25:13,023 INFO [org.ovirt.engine.core.dal.job.ExecutionMessageDirector] (ServerService Thread Pool -- 51) Start initializing ExecutionMessageDirector >2014-09-09 15:25:13,025 INFO [org.ovirt.engine.core.dal.job.ExecutionMessageDirector] (ServerService Thread Pool -- 51) Finished initializing ExecutionMessageDirector >2014-09-09 15:25:13,136 INFO [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (ServerService Thread Pool -- 51) Loading extension 'builtin-authn-internal' >2014-09-09 15:25:13,138 INFO [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (ServerService Thread Pool -- 51) Extension 'builtin-authn-internal' loaded >2014-09-09 15:25:13,139 INFO [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (ServerService Thread Pool -- 51) Loading extension 'internal' >2014-09-09 15:25:13,147 INFO [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (ServerService Thread Pool -- 51) Extension 'internal' loaded >2014-09-09 15:25:13,149 INFO [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (ServerService Thread Pool -- 51) Initializing extension 'builtin-authn-internal' >2014-09-09 15:25:13,149 INFO [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (ServerService Thread Pool -- 51) Extension 'builtin-authn-internal' initialized >2014-09-09 15:25:13,150 INFO [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (ServerService Thread Pool -- 51) Initializing extension 'internal' >2014-09-09 15:25:13,150 INFO [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (ServerService Thread Pool -- 51) Extension 'internal' initialized >2014-09-09 15:25:13,150 INFO [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (ServerService Thread Pool -- 51) Start of enabled extensions list >2014-09-09 15:25:13,150 INFO [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (ServerService Thread Pool -- 51) Instance name: 'builtin-authn-internal', Extension name: 'Internal Authn (Built-in)', Version: 'N/A', Notes: '', License: 'ASL 2.0', Home: 'http://www.ovirt.org', Author 'The oVirt Project', Build interface Version: '0', File: 'N/A', Initialized: 'true' >2014-09-09 15:25:13,150 INFO [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (ServerService Thread Pool -- 51) Instance name: 'internal', Extension name: 'Internal Authz (Built-in)', Version: 'N/A', Notes: '', License: 'ASL 2.0', Home: 'http://www.ovirt.org', Author 'The oVirt Project', Build interface Version: '0', File: 'N/A', Initialized: 'true' >2014-09-09 15:25:13,150 INFO [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (ServerService Thread Pool -- 51) End of enabled extensions list >2014-09-09 15:25:13,275 INFO [org.ovirt.engine.core.bll.aaa.DbUserCacheManager] (ServerService Thread Pool -- 48) Start initializing DbUserCacheManager >2014-09-09 15:25:13,278 INFO [org.ovirt.engine.core.bll.aaa.DbUserCacheManager] (ServerService Thread Pool -- 48) Finished initializing DbUserCacheManager >2014-09-09 15:25:13,332 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (ServerService Thread Pool -- 48) Initialization of AsyncTaskManager completed successfully. >2014-09-09 15:25:13,334 INFO [org.ovirt.engine.core.vdsbroker.ResourceManager] (ServerService Thread Pool -- 48) Start initializing ResourceManager >2014-09-09 15:25:13,394 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-1) Principal admin::internal synchronized >2014-09-09 15:25:13,579 INFO [org.ovirt.engine.core.vdsbroker.ResourceManager] (ServerService Thread Pool -- 48) Finished initializing ResourceManager >2014-09-09 15:25:13,584 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (ServerService Thread Pool -- 48) Initialization of OvfDataUpdater completed successfully. >2014-09-09 15:25:13,585 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (ServerService Thread Pool -- 48) Start scheduling to enable vds load balancer >2014-09-09 15:25:13,585 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (ServerService Thread Pool -- 48) Finished scheduling to enable vds load balancer >2014-09-09 15:25:13,585 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (ServerService Thread Pool -- 48) Start HA Reservation check >2014-09-09 15:25:13,586 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (ServerService Thread Pool -- 48) Finished HA Reservation check >2014-09-09 15:25:13,600 INFO [org.ovirt.engine.core.bll.network.MacPoolManagerRanges] (org.ovirt.thread.pool-6-thread-1) Start initializing MacPoolManagerRanges >2014-09-09 15:25:13,610 INFO [org.ovirt.engine.core.bll.InitBackendServicesOnStartupBean] (ServerService Thread Pool -- 48) Init VM custom properties utilities >2014-09-09 15:25:13,611 INFO [org.ovirt.engine.core.bll.InitBackendServicesOnStartupBean] (ServerService Thread Pool -- 48) Init device custom properties utilities >2014-09-09 15:25:13,621 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (ServerService Thread Pool -- 48) Initializing Scheduling manager >2014-09-09 15:25:13,649 INFO [org.ovirt.engine.core.bll.network.MacPoolManagerRanges] (org.ovirt.thread.pool-6-thread-1) Finished initializing. Available MACs in pool: 256 >2014-09-09 15:25:13,684 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (ServerService Thread Pool -- 48) External scheduler disabled, discovery skipped >2014-09-09 15:25:13,684 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (ServerService Thread Pool -- 48) Initialized Scheduling manager >2014-09-09 15:25:13,684 INFO [org.ovirt.engine.core.bll.dwh.DwhHeartBeat] (ServerService Thread Pool -- 48) Initializing DWH Heart Beat >2014-09-09 15:25:13,692 INFO [org.ovirt.engine.core.bll.dwh.DwhHeartBeat] (ServerService Thread Pool -- 48) DWH Heart Beat initialized >2014-09-09 15:28:26,703 INFO [org.ovirt.engine.core.utils.servlet.UnsupportedLocaleHelper] (ajp-/127.0.0.1:8702-1) Invalid locale found in configuration: >2014-09-09 15:28:26,707 INFO [org.ovirt.engine.core.utils.servlet.UnsupportedLocaleHelper] (ajp-/127.0.0.1:8702-1) Invalid locale found in configuration: >2014-09-09 15:28:38,980 INFO [org.ovirt.engine.ui.frontend.server.gwt.plugin.PluginDataManager] (ajp-/127.0.0.1:8702-1) Reading UI plugin descriptor [/usr/share/ovirt-engine/ui-plugins/redhat_support_plugin_rhev.json] >2014-09-09 15:28:38,996 INFO [org.ovirt.engine.ui.frontend.server.gwt.plugin.PluginDataManager] (ajp-/127.0.0.1:8702-1) Reading UI plugin configuration [/etc/ovirt-engine/ui-plugins/redhat_support_plugin_rhev-config.json] >2014-09-09 15:28:39,023 INFO [org.ovirt.engine.core.utils.servlet.UnsupportedLocaleHelper] (ajp-/127.0.0.1:8702-1) Invalid locale found in configuration: >2014-09-09 15:28:39,023 INFO [org.ovirt.engine.core.utils.servlet.UnsupportedLocaleHelper] (ajp-/127.0.0.1:8702-1) Invalid locale found in configuration: >2014-09-09 15:28:58,966 INFO [org.ovirt.engine.core.bll.aaa.LoginAdminUserCommand] (ajp-/127.0.0.1:8702-4) Running command: LoginAdminUserCommand internal: false. >2014-09-09 15:28:58,980 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-4) AuditLogType: UNASSIGNED not exist in string table >2014-09-09 15:28:58,981 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-4) AuditLogType: VDS_AUTO_FENCE_STATUS not exist in string table >2014-09-09 15:28:58,981 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-4) AuditLogType: VDS_AUTO_FENCE_STATUS_FAILED not exist in string table >2014-09-09 15:28:58,981 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-4) AuditLogType: VDS_AUTO_FENCE_FAILED_CALL_FENCE_SPM not exist in string table >2014-09-09 15:28:58,981 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-4) AuditLogType: VDS_HIGH_NETWORK_USE not exist in string table >2014-09-09 15:28:58,985 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-4) AuditLogType: USER_FAILED_REMOVE_VM not exist in string table >2014-09-09 15:28:58,985 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-4) AuditLogType: USER_RUN_UNLOCK_ENTITY_SCRIPT not exist in string table >2014-09-09 15:28:58,986 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-4) AuditLogType: VDS_NETWORK_MTU_DIFFER_FROM_LOGICAL_NETWORK not exist in string table >2014-09-09 15:28:58,986 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-4) AuditLogType: STORAGE_ACTIVATE_ASYNC not exist in string table >2014-09-09 15:28:58,986 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-4) AuditLogType: DWH_STOPPED not exist in string table >2014-09-09 15:28:58,986 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-4) AuditLogType: DWH_STARTED not exist in string table >2014-09-09 15:28:58,986 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-4) AuditLogType: DWH_ERROR not exist in string table >2014-09-09 15:28:58,987 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-4) AuditLogType: USER_ADDED_DISK_PROFILE not exist in string table >2014-09-09 15:28:58,987 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-4) AuditLogType: USER_FAILED_TO_ADD_DISK_PROFILE not exist in string table >2014-09-09 15:28:58,987 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-4) AuditLogType: USER_REMOVED_DISK_PROFILE not exist in string table >2014-09-09 15:28:58,987 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-4) AuditLogType: USER_FAILED_TO_REMOVE_DISK_PROFILE not exist in string table >2014-09-09 15:28:58,987 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-4) AuditLogType: USER_UPDATED_DISK_PROFILE not exist in string table >2014-09-09 15:28:58,987 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-4) AuditLogType: USER_FAILED_TO_UPDATE_DISK_PROFILE not exist in string table >2014-09-09 15:28:58,987 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-4) AuditLogType: USER_ADDED_CPU_PROFILE not exist in string table >2014-09-09 15:28:58,989 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-4) AuditLogType: USER_FAILED_TO_ADD_CPU_PROFILE not exist in string table >2014-09-09 15:28:58,989 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-4) AuditLogType: USER_REMOVED_CPU_PROFILE not exist in string table >2014-09-09 15:28:58,989 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-4) AuditLogType: USER_FAILED_TO_REMOVE_CPU_PROFILE not exist in string table >2014-09-09 15:28:58,991 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-4) AuditLogType: USER_UPDATED_CPU_PROFILE not exist in string table >2014-09-09 15:28:58,992 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-4) AuditLogType: USER_FAILED_TO_UPDATE_CPU_PROFILE not exist in string table >2014-09-09 15:28:58,992 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-4) AuditLogType: USER_REMOVE_AUDIT_LOG not exist in string table >2014-09-09 15:28:58,992 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-4) AuditLogType: USER_REMOVE_AUDIT_LOG_FAILED not exist in string table >2014-09-09 15:28:58,992 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-4) AuditLogType: USER_CLEAR_ALL_DISMISSED_AUDIT_LOG not exist in string table >2014-09-09 15:28:58,992 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-4) AuditLogType: USER_CLEAR_ALL_DISMISSED_AUDIT_LOG_FAILED not exist in string table >2014-09-09 15:28:59,494 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-4) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-09-09 15:29:04,235 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp-/127.0.0.1:8702-2) Running command: LoginUserCommand internal: false. >2014-09-09 15:31:48,934 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Loaded file "/usr/share/ovirt-engine/services/ovirt-engine/ovirt-engine.conf". >2014-09-09 15:31:48,934 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) The file "/etc/ovirt-engine/engine.conf" doesn't exist or isn't readable. Will return an empty set of properties. >2014-09-09 15:31:48,935 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Loaded file "/etc/ovirt-engine/engine.conf.d/10-rhevm-doc.conf". >2014-09-09 15:31:48,935 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Loaded file "/etc/ovirt-engine/engine.conf.d/10-setup-database.conf". >2014-09-09 15:31:48,935 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Loaded file "/etc/ovirt-engine/engine.conf.d/10-setup-jboss.conf". >2014-09-09 15:31:48,944 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Loaded file "/etc/ovirt-engine/engine.conf.d/10-setup-pki.conf". >2014-09-09 15:31:48,948 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Loaded file "/etc/ovirt-engine/engine.conf.d/10-setup-protocols.conf". >2014-09-09 15:31:48,948 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Loaded file "/etc/ovirt-engine/engine.conf.d/20-branding-rhev.conf". >2014-09-09 15:31:48,948 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Loaded file "/etc/ovirt-engine/engine.conf.d/50-redhat-support-plugin.conf". >2014-09-09 15:31:48,949 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Value of property "ENGINE_AJP_ENABLED" is "true". >2014-09-09 15:31:48,949 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Value of property "ENGINE_AJP_PORT" is "8702". >2014-09-09 15:31:48,949 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Value of property "ENGINE_APPS" is "engine.ear restapi.war legacy_restapi.war "/usr/share/ovirt-engine/branding/rhev-2.brand/applications/rhev.ear" redhat-support-plugin-rhev.ear". >2014-09-09 15:31:48,949 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Value of property "ENGINE_CACHE" is "/var/cache/ovirt-engine". >2014-09-09 15:31:48,949 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Value of property "ENGINE_DB_CHECK_INTERVAL" is "1000". >2014-09-09 15:31:48,949 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Value of property "ENGINE_DB_CONNECTION_TIMEOUT" is "300000". >2014-09-09 15:31:48,949 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Value of property "ENGINE_DB_DATABASE" is "engine". >2014-09-09 15:31:48,949 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Value of property "ENGINE_DB_DRIVER" is "org.postgresql.Driver". >2014-09-09 15:31:48,949 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Value of property "ENGINE_DB_HOST" is "localhost". >2014-09-09 15:31:48,950 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Value of property "ENGINE_DB_MAX_CONNECTIONS" is "100". >2014-09-09 15:31:48,950 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Value of property "ENGINE_DB_MIN_CONNECTIONS" is "1". >2014-09-09 15:31:48,950 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Value of property "ENGINE_DB_PASSWORD" is "***". >2014-09-09 15:31:48,950 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Value of property "ENGINE_DB_PORT" is "5432". >2014-09-09 15:31:48,950 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Value of property "ENGINE_DB_SECURED" is "False". >2014-09-09 15:31:48,950 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Value of property "ENGINE_DB_SECURED_VALIDATION" is "False". >2014-09-09 15:31:48,950 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Value of property "ENGINE_DB_URL" is "jdbc:postgresql://localhost:5432/engine?sslfactory=org.postgresql.ssl.NonValidatingFactory". >2014-09-09 15:31:48,950 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Value of property "ENGINE_DB_USER" is "engine". >2014-09-09 15:31:48,950 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Value of property "ENGINE_DEBUG_ADDRESS" is "". >2014-09-09 15:31:48,951 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Value of property "ENGINE_DOC" is "/usr/share/doc/ovirt-engine". >2014-09-09 15:31:48,951 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Value of property "ENGINE_ETC" is "/etc/ovirt-engine". >2014-09-09 15:31:48,951 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Value of property "ENGINE_EXTENSION_PATH" is "/usr/share/ovirt-engine/extensions.d:/etc/ovirt-engine/extensions.d". >2014-09-09 15:31:48,951 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Value of property "ENGINE_FQDN" is "pb-rh35.rhev.lab.eng.brq.redhat.com". >2014-09-09 15:31:48,951 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Value of property "ENGINE_GROUP" is "ovirt". >2014-09-09 15:31:48,951 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Value of property "ENGINE_HEAP_MAX" is "1g". >2014-09-09 15:31:48,951 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Value of property "ENGINE_HEAP_MIN" is "1g". >2014-09-09 15:31:48,960 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Value of property "ENGINE_HTTPS_ENABLED" is "false". >2014-09-09 15:31:48,960 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Value of property "ENGINE_HTTPS_PORT" is "None". >2014-09-09 15:31:48,960 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Value of property "ENGINE_HTTPS_PROTOCOLS" is "SSLv3,TLSv1,TLSv1.1,TLSv1.2". >2014-09-09 15:31:48,960 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Value of property "ENGINE_HTTP_ENABLED" is "false". >2014-09-09 15:31:48,960 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Value of property "ENGINE_HTTP_PORT" is "None". >2014-09-09 15:31:48,960 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Value of property "ENGINE_JAVA_MODULEPATH" is "/usr/share/ovirt-engine/modules". >2014-09-09 15:31:48,960 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Value of property "ENGINE_JVM_ARGS" is " -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath="/var/log/ovirt-engine/dump"". >2014-09-09 15:31:48,961 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Value of property "ENGINE_LOG" is "/var/log/ovirt-engine". >2014-09-09 15:31:48,961 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Value of property "ENGINE_LOG_TO_CONSOLE" is "false". >2014-09-09 15:31:48,961 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Value of property "ENGINE_MANUAL" is "/usr/share/doc/rhevm-doc/manual". >2014-09-09 15:31:48,961 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Value of property "ENGINE_PERM_MAX" is "256m". >2014-09-09 15:31:48,961 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Value of property "ENGINE_PERM_MIN" is "256m". >2014-09-09 15:31:48,961 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Value of property "ENGINE_PKI" is "/etc/pki/ovirt-engine". >2014-09-09 15:31:48,961 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Value of property "ENGINE_PKI_CA" is "/etc/pki/ovirt-engine/ca.pem". >2014-09-09 15:31:48,961 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Value of property "ENGINE_PKI_ENGINE_CERT" is "/etc/pki/ovirt-engine/certs/engine.cer". >2014-09-09 15:31:48,961 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Value of property "ENGINE_PKI_ENGINE_STORE" is "/etc/pki/ovirt-engine/keys/engine.p12". >2014-09-09 15:31:48,962 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Value of property "ENGINE_PKI_ENGINE_STORE_ALIAS" is "1". >2014-09-09 15:31:48,962 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Value of property "ENGINE_PKI_ENGINE_STORE_PASSWORD" is "***". >2014-09-09 15:31:48,962 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Value of property "ENGINE_PKI_TRUST_STORE" is "/etc/pki/ovirt-engine/.truststore". >2014-09-09 15:31:48,962 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Value of property "ENGINE_PKI_TRUST_STORE_PASSWORD" is "***". >2014-09-09 15:31:48,962 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Value of property "ENGINE_PROPERTIES" is " jsse.enableSNIExtension=false "java.security.krb5.conf=/etc/ovirt-engine/krb5.conf"". >2014-09-09 15:31:48,962 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Value of property "ENGINE_PROXY_ENABLED" is "true". >2014-09-09 15:31:48,962 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Value of property "ENGINE_PROXY_HTTPS_PORT" is "443". >2014-09-09 15:31:48,962 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Value of property "ENGINE_PROXY_HTTP_PORT" is "80". >2014-09-09 15:31:48,962 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Value of property "ENGINE_REPORTS_UI" is "/var/lib/ovirt-engine/reports.xml". >2014-09-09 15:31:48,963 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Value of property "ENGINE_STOP_INTERVAL" is "1". >2014-09-09 15:31:48,963 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Value of property "ENGINE_STOP_TIME" is "10". >2014-09-09 15:31:48,963 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Value of property "ENGINE_TMP" is "/var/tmp/ovirt-engine". >2014-09-09 15:31:48,963 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Value of property "ENGINE_UP_MARK" is "/var/lib/ovirt-engine/engine.up". >2014-09-09 15:31:48,963 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Value of property "ENGINE_URI" is "/ovirt-engine". >2014-09-09 15:31:48,963 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Value of property "ENGINE_USER" is "ovirt". >2014-09-09 15:31:48,963 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Value of property "ENGINE_USR" is "/usr/share/ovirt-engine". >2014-09-09 15:31:48,963 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Value of property "ENGINE_VAR" is "/var/lib/ovirt-engine". >2014-09-09 15:31:48,971 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Value of property "ENGINE_VERBOSE_GC" is "false". >2014-09-09 15:31:48,972 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Value of property "JBOSS_HOME" is "/usr/share/jbossas". >2014-09-09 15:31:48,972 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Value of property "JBOSS_RUNTIME" is "/var/lib/ovirt-engine/jboss_runtime". >2014-09-09 15:31:48,972 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Value of property "SENSITIVE_KEYS" is ",ENGINE_DB_PASSWORD,ENGINE_PKI_TRUST_STORE_PASSWORD,ENGINE_PKI_ENGINE_STORE_PASSWORD". >2014-09-09 15:31:59,286 INFO [org.ovirt.engine.core.bll.Backend] (ajp-/127.0.0.1:8702-1) Start initializing Backend >2014-09-09 15:31:59,698 INFO [org.ovirt.engine.core.utils.osinfo.OsInfoPreferencesLoader] (ajp-/127.0.0.1:8702-1) Loaded file /etc/ovirt-engine/osinfo.conf.d/00-defaults.properties >2014-09-09 15:31:59,818 INFO [org.ovirt.engine.core.bll.Backend] (ajp-/127.0.0.1:8702-1) Running ovirt-engine 3.5.0-0.10.master.el6ev >2014-09-09 15:31:59,818 INFO [org.ovirt.engine.core.bll.CpuFlagsManagerHandler] (ajp-/127.0.0.1:8702-1) Start initializing dictionaries >2014-09-09 15:31:59,826 INFO [org.ovirt.engine.core.bll.CpuFlagsManagerHandler] (ajp-/127.0.0.1:8702-1) Finished initializing dictionaries >2014-09-09 15:31:59,826 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (ajp-/127.0.0.1:8702-1) Start initializing AuditLogCleanupManager >2014-09-09 15:31:59,826 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (ajp-/127.0.0.1:8702-1) Setting audit cleanup manager to run at: 35 35 3 * * ? >2014-09-09 15:31:59,844 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (ajp-/127.0.0.1:8702-1) Finished initializing AuditLogCleanupManager >2014-09-09 15:31:59,845 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (ajp-/127.0.0.1:8702-1) Start initializing CommandEntityCleanupManager >2014-09-09 15:31:59,845 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (ajp-/127.0.0.1:8702-1) Setting command entity cleanup manager to run at: 35 35 3 * * ? >2014-09-09 15:31:59,850 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (ajp-/127.0.0.1:8702-1) Finished initializing CommandEntityCleanupManager >2014-09-09 15:31:59,853 INFO [org.ovirt.engine.core.bll.TagsDirector] (ajp-/127.0.0.1:8702-1) Start initializing TagsDirector >2014-09-09 15:31:59,866 INFO [org.ovirt.engine.core.bll.TagsDirector] (ajp-/127.0.0.1:8702-1) Tag root added to tree >2014-09-09 15:31:59,884 INFO [org.ovirt.engine.core.bll.TagsDirector] (ajp-/127.0.0.1:8702-1) Finished initializing TagsDirector >2014-09-09 15:31:59,885 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (ajp-/127.0.0.1:8702-1) Start initializing IsoDomainListSyncronizer >2014-09-09 15:31:59,891 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (ajp-/127.0.0.1:8702-1) Finished initializing IsoDomainListSyncronizer >2014-09-09 15:31:59,937 INFO [org.ovirt.engine.core.bll.Backend] (ajp-/127.0.0.1:8702-1) Completed initializing handlers >2014-09-09 15:31:59,965 INFO [org.ovirt.engine.core.utils.ErrorTranslatorImpl] (ajp-/127.0.0.1:8702-1) Start initializing ErrorTranslatorImpl >2014-09-09 15:31:59,984 WARN [org.ovirt.engine.core.utils.ErrorTranslatorImpl] (ajp-/127.0.0.1:8702-1) Code MAC_ADDRESS_IS_IN_USE appears more than once in string table. >2014-09-09 15:31:59,985 INFO [org.ovirt.engine.core.utils.ErrorTranslatorImpl] (ajp-/127.0.0.1:8702-1) Finished initializing ErrorTranslatorImpl >2014-09-09 15:31:59,985 INFO [org.ovirt.engine.core.utils.ErrorTranslatorImpl] (ajp-/127.0.0.1:8702-1) Start initializing ErrorTranslatorImpl >2014-09-09 15:31:59,986 INFO [org.ovirt.engine.core.utils.ErrorTranslatorImpl] (ajp-/127.0.0.1:8702-1) Finished initializing ErrorTranslatorImpl >2014-09-09 15:31:59,986 INFO [org.ovirt.engine.core.bll.Backend] (ajp-/127.0.0.1:8702-1) Mark incomplete jobs as UNKNOWN >2014-09-09 15:32:00,029 INFO [org.ovirt.engine.core.bll.job.JobRepositoryCleanupManager] (ajp-/127.0.0.1:8702-1) Start initializing JobRepositoryCleanupManager >2014-09-09 15:32:00,030 INFO [org.ovirt.engine.core.bll.job.JobRepositoryCleanupManager] (ajp-/127.0.0.1:8702-1) Finished initializing JobRepositoryCleanupManager >2014-09-09 15:32:00,030 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (ajp-/127.0.0.1:8702-1) Start initializing AutoRecoveryManager >2014-09-09 15:32:00,034 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (ajp-/127.0.0.1:8702-1) Finished initializing AutoRecoveryManager >2014-09-09 15:32:00,036 INFO [org.ovirt.engine.core.dal.job.ExecutionMessageDirector] (ajp-/127.0.0.1:8702-1) Start initializing ExecutionMessageDirector >2014-09-09 15:32:00,038 INFO [org.ovirt.engine.core.dal.job.ExecutionMessageDirector] (ajp-/127.0.0.1:8702-1) Finished initializing ExecutionMessageDirector >2014-09-09 15:32:00,155 INFO [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (ajp-/127.0.0.1:8702-1) Loading extension 'builtin-authn-internal' >2014-09-09 15:32:00,158 INFO [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (ajp-/127.0.0.1:8702-1) Extension 'builtin-authn-internal' loaded >2014-09-09 15:32:00,161 INFO [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (ajp-/127.0.0.1:8702-1) Loading extension 'internal' >2014-09-09 15:32:00,164 INFO [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (ajp-/127.0.0.1:8702-1) Extension 'internal' loaded >2014-09-09 15:32:00,164 INFO [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (ajp-/127.0.0.1:8702-1) Initializing extension 'builtin-authn-internal' >2014-09-09 15:32:00,164 INFO [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (ajp-/127.0.0.1:8702-1) Extension 'builtin-authn-internal' initialized >2014-09-09 15:32:00,165 INFO [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (ajp-/127.0.0.1:8702-1) Initializing extension 'internal' >2014-09-09 15:32:00,165 INFO [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (ajp-/127.0.0.1:8702-1) Extension 'internal' initialized >2014-09-09 15:32:00,166 INFO [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (ajp-/127.0.0.1:8702-1) Start of enabled extensions list >2014-09-09 15:32:00,166 INFO [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (ajp-/127.0.0.1:8702-1) Instance name: 'builtin-authn-internal', Extension name: 'Internal Authn (Built-in)', Version: 'N/A', Notes: '', License: 'ASL 2.0', Home: 'http://www.ovirt.org', Author 'The oVirt Project', Build interface Version: '0', File: 'N/A', Initialized: 'true' >2014-09-09 15:32:00,166 INFO [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (ajp-/127.0.0.1:8702-1) Instance name: 'internal', Extension name: 'Internal Authz (Built-in)', Version: 'N/A', Notes: '', License: 'ASL 2.0', Home: 'http://www.ovirt.org', Author 'The oVirt Project', Build interface Version: '0', File: 'N/A', Initialized: 'true' >2014-09-09 15:32:00,166 INFO [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (ajp-/127.0.0.1:8702-1) End of enabled extensions list >2014-09-09 15:32:00,508 INFO [org.ovirt.engine.core.bll.aaa.DbUserCacheManager] (ServerService Thread Pool -- 44) Start initializing DbUserCacheManager >2014-09-09 15:32:00,512 INFO [org.ovirt.engine.core.bll.aaa.DbUserCacheManager] (ServerService Thread Pool -- 44) Finished initializing DbUserCacheManager >2014-09-09 15:32:00,566 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (ServerService Thread Pool -- 44) Initialization of AsyncTaskManager completed successfully. >2014-09-09 15:32:00,567 INFO [org.ovirt.engine.core.vdsbroker.ResourceManager] (ServerService Thread Pool -- 44) Start initializing ResourceManager >2014-09-09 15:32:00,676 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-1) Principal admin::internal synchronized >2014-09-09 15:32:00,815 INFO [org.ovirt.engine.core.vdsbroker.ResourceManager] (ServerService Thread Pool -- 44) Finished initializing ResourceManager >2014-09-09 15:32:00,820 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (ServerService Thread Pool -- 44) Initialization of OvfDataUpdater completed successfully. >2014-09-09 15:32:00,821 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (ServerService Thread Pool -- 44) Start scheduling to enable vds load balancer >2014-09-09 15:32:00,821 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (ServerService Thread Pool -- 44) Finished scheduling to enable vds load balancer >2014-09-09 15:32:00,821 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (ServerService Thread Pool -- 44) Start HA Reservation check >2014-09-09 15:32:00,821 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (ServerService Thread Pool -- 44) Finished HA Reservation check >2014-09-09 15:32:00,839 INFO [org.ovirt.engine.core.bll.network.MacPoolManagerRanges] (org.ovirt.thread.pool-6-thread-1) Start initializing MacPoolManagerRanges >2014-09-09 15:32:00,855 INFO [org.ovirt.engine.core.bll.InitBackendServicesOnStartupBean] (ServerService Thread Pool -- 44) Init VM custom properties utilities >2014-09-09 15:32:00,856 INFO [org.ovirt.engine.core.bll.InitBackendServicesOnStartupBean] (ServerService Thread Pool -- 44) Init device custom properties utilities >2014-09-09 15:32:00,866 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (ServerService Thread Pool -- 44) Initializing Scheduling manager >2014-09-09 15:32:00,887 INFO [org.ovirt.engine.core.bll.network.MacPoolManagerRanges] (org.ovirt.thread.pool-6-thread-1) Finished initializing. Available MACs in pool: 65 >2014-09-09 15:32:00,951 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (ServerService Thread Pool -- 44) External scheduler disabled, discovery skipped >2014-09-09 15:32:00,951 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (ServerService Thread Pool -- 44) Initialized Scheduling manager >2014-09-09 15:32:00,951 INFO [org.ovirt.engine.core.bll.dwh.DwhHeartBeat] (ServerService Thread Pool -- 44) Initializing DWH Heart Beat >2014-09-09 15:32:00,960 INFO [org.ovirt.engine.core.bll.dwh.DwhHeartBeat] (ServerService Thread Pool -- 44) DWH Heart Beat initialized >2014-09-09 15:32:02,460 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp-/127.0.0.1:8702-4) [3225340d] Running command: LogoutUserCommand internal: false. >2014-09-09 15:32:02,554 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-4) [3225340d] AuditLogType: UNASSIGNED not exist in string table >2014-09-09 15:32:02,554 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-4) [3225340d] AuditLogType: VDS_AUTO_FENCE_STATUS not exist in string table >2014-09-09 15:32:02,554 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-4) [3225340d] AuditLogType: VDS_AUTO_FENCE_STATUS_FAILED not exist in string table >2014-09-09 15:32:02,555 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-4) [3225340d] AuditLogType: VDS_AUTO_FENCE_FAILED_CALL_FENCE_SPM not exist in string table >2014-09-09 15:32:02,555 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-4) [3225340d] AuditLogType: VDS_HIGH_NETWORK_USE not exist in string table >2014-09-09 15:32:02,555 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-4) [3225340d] AuditLogType: USER_FAILED_REMOVE_VM not exist in string table >2014-09-09 15:32:02,555 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-4) [3225340d] AuditLogType: USER_RUN_UNLOCK_ENTITY_SCRIPT not exist in string table >2014-09-09 15:32:02,556 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-4) [3225340d] AuditLogType: VDS_NETWORK_MTU_DIFFER_FROM_LOGICAL_NETWORK not exist in string table >2014-09-09 15:32:02,557 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-4) [3225340d] AuditLogType: STORAGE_ACTIVATE_ASYNC not exist in string table >2014-09-09 15:32:02,557 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-4) [3225340d] AuditLogType: DWH_STOPPED not exist in string table >2014-09-09 15:32:02,557 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-4) [3225340d] AuditLogType: DWH_STARTED not exist in string table >2014-09-09 15:32:02,557 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-4) [3225340d] AuditLogType: DWH_ERROR not exist in string table >2014-09-09 15:32:02,558 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-4) [3225340d] AuditLogType: USER_ADDED_DISK_PROFILE not exist in string table >2014-09-09 15:32:02,558 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-4) [3225340d] AuditLogType: USER_FAILED_TO_ADD_DISK_PROFILE not exist in string table >2014-09-09 15:32:02,558 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-4) [3225340d] AuditLogType: USER_REMOVED_DISK_PROFILE not exist in string table >2014-09-09 15:32:02,558 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-4) [3225340d] AuditLogType: USER_FAILED_TO_REMOVE_DISK_PROFILE not exist in string table >2014-09-09 15:32:02,558 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-4) [3225340d] AuditLogType: USER_UPDATED_DISK_PROFILE not exist in string table >2014-09-09 15:32:02,558 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-4) [3225340d] AuditLogType: USER_FAILED_TO_UPDATE_DISK_PROFILE not exist in string table >2014-09-09 15:32:02,559 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-4) [3225340d] AuditLogType: USER_ADDED_CPU_PROFILE not exist in string table >2014-09-09 15:32:02,559 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-4) [3225340d] AuditLogType: USER_FAILED_TO_ADD_CPU_PROFILE not exist in string table >2014-09-09 15:32:02,559 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-4) [3225340d] AuditLogType: USER_REMOVED_CPU_PROFILE not exist in string table >2014-09-09 15:32:02,559 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-4) [3225340d] AuditLogType: USER_FAILED_TO_REMOVE_CPU_PROFILE not exist in string table >2014-09-09 15:32:02,559 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-4) [3225340d] AuditLogType: USER_UPDATED_CPU_PROFILE not exist in string table >2014-09-09 15:32:02,559 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-4) [3225340d] AuditLogType: USER_FAILED_TO_UPDATE_CPU_PROFILE not exist in string table >2014-09-09 15:32:02,560 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-4) [3225340d] AuditLogType: USER_REMOVE_AUDIT_LOG not exist in string table >2014-09-09 15:32:02,560 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-4) [3225340d] AuditLogType: USER_REMOVE_AUDIT_LOG_FAILED not exist in string table >2014-09-09 15:32:02,560 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-4) [3225340d] AuditLogType: USER_CLEAR_ALL_DISMISSED_AUDIT_LOG not exist in string table >2014-09-09 15:32:02,561 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-4) [3225340d] AuditLogType: USER_CLEAR_ALL_DISMISSED_AUDIT_LOG_FAILED not exist in string table >2014-09-09 15:32:02,673 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-4) [3225340d] Correlation ID: 3225340d, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-09-09 15:32:03,072 INFO [org.ovirt.engine.ui.frontend.server.gwt.plugin.PluginDataManager] (ajp-/127.0.0.1:8702-9) Reading UI plugin descriptor [/usr/share/ovirt-engine/ui-plugins/redhat_support_plugin_rhev.json] >2014-09-09 15:32:03,094 INFO [org.ovirt.engine.ui.frontend.server.gwt.plugin.PluginDataManager] (ajp-/127.0.0.1:8702-9) Reading UI plugin configuration [/etc/ovirt-engine/ui-plugins/redhat_support_plugin_rhev-config.json] >2014-09-09 15:32:03,428 INFO [org.ovirt.engine.core.utils.servlet.UnsupportedLocaleHelper] (ajp-/127.0.0.1:8702-9) Invalid locale found in configuration: >2014-09-09 15:32:03,428 INFO [org.ovirt.engine.core.utils.servlet.UnsupportedLocaleHelper] (ajp-/127.0.0.1:8702-9) Invalid locale found in configuration: >2014-09-09 15:32:16,342 INFO [org.ovirt.engine.core.bll.aaa.LoginAdminUserCommand] (ajp-/127.0.0.1:8702-13) Running command: LoginAdminUserCommand internal: false. >2014-09-09 15:32:16,770 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-13) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-09-09 15:32:20,519 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp-/127.0.0.1:8702-12) Running command: LoginUserCommand internal: false. >2014-09-09 15:36:57,175 WARN [org.ovirt.engine.core.bll.AddVdsCommand] (ajp-/127.0.0.1:8702-1) [14519643] CanDoAction of action AddVds failed. Reasons:VAR__ACTION__ADD,VAR__TYPE__HOST,$server dell-r210ii-16.rhev.lab.eng.brq.redhat.com,VDS_CANNOT_INSTALL_EMPTY_PASSWORD >2014-09-09 15:37:08,173 INFO [org.ovirt.engine.core.bll.AddVdsCommand] (ajp-/127.0.0.1:8702-4) [3307426a] Running command: AddVdsCommand internal: false. Entities affected : ID: 00000001-0001-0001-0001-000000000399 Type: VdsGroupsAction group CREATE_HOST with role type ADMIN >2014-09-09 15:37:08,500 INFO [org.ovirt.engine.core.bll.AddVdsSpmIdCommand] (ajp-/127.0.0.1:8702-4) [5cb487dd] Running command: AddVdsSpmIdCommand internal: true. Entities affected : ID: 14369bd0-907d-40d7-be1e-bd35af6d1488 Type: VDS >2014-09-09 15:37:08,530 INFO [org.ovirt.engine.core.bll.AddVdsSpmIdCommand] (ajp-/127.0.0.1:8702-4) [5cb487dd] Lock freed to object EngineLock [exclusiveLocks= key: 00000002-0002-0002-0002-0000000001ad value: REGISTER_VDS >, sharedLocks= ] >2014-09-09 15:37:08,540 INFO [org.ovirt.engine.core.vdsbroker.RemoveVdsVDSCommand] (ajp-/127.0.0.1:8702-4) [5cb487dd] START, RemoveVdsVDSCommand(HostName = rack, HostId = 14369bd0-907d-40d7-be1e-bd35af6d1488), log id: 712d6003 >2014-09-09 15:37:08,541 INFO [org.ovirt.engine.core.vdsbroker.RemoveVdsVDSCommand] (ajp-/127.0.0.1:8702-4) [5cb487dd] FINISH, RemoveVdsVDSCommand, log id: 712d6003 >2014-09-09 15:37:08,545 INFO [org.ovirt.engine.core.vdsbroker.AddVdsVDSCommand] (ajp-/127.0.0.1:8702-4) [5cb487dd] START, AddVdsVDSCommand(HostName = rack, HostId = 14369bd0-907d-40d7-be1e-bd35af6d1488), log id: 4c5eff5a >2014-09-09 15:37:08,545 INFO [org.ovirt.engine.core.vdsbroker.AddVdsVDSCommand] (ajp-/127.0.0.1:8702-4) [5cb487dd] AddVds - entered , starting logic to add VDS 14369bd0-907d-40d7-be1e-bd35af6d1488 >2014-09-09 15:37:08,548 INFO [org.ovirt.engine.core.vdsbroker.AddVdsVDSCommand] (ajp-/127.0.0.1:8702-4) [5cb487dd] AddVds - VDS 14369bd0-907d-40d7-be1e-bd35af6d1488 was added, will try to add it to the resource manager >2014-09-09 15:37:08,553 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (ajp-/127.0.0.1:8702-4) [5cb487dd] Entered VdsManager constructor >2014-09-09 15:37:08,580 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (ajp-/127.0.0.1:8702-4) [5cb487dd] Initialize vdsBroker (dell-r210ii-16.rhev.lab.eng.brq.redhat.com,54,321) >2014-09-09 15:37:08,748 INFO [org.ovirt.engine.core.vdsbroker.ResourceManager] (ajp-/127.0.0.1:8702-4) [5cb487dd] VDS 14369bd0-907d-40d7-be1e-bd35af6d1488 was added to the Resource Manager >2014-09-09 15:37:08,757 INFO [org.ovirt.engine.core.vdsbroker.AddVdsVDSCommand] (ajp-/127.0.0.1:8702-4) [5cb487dd] FINISH, AddVdsVDSCommand, log id: 4c5eff5a >2014-09-09 15:37:08,774 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-4) [5cb487dd] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host rack. >2014-09-09 15:37:08,848 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-4) [5cb487dd] Correlation ID: 3307426a, Job ID: a335eda2-f1e7-4c42-a7c8-1714f9c188bd, Call Stack: null, Custom Event ID: -1, Message: Host rack was added by admin. >2014-09-09 15:37:08,895 INFO [org.ovirt.engine.core.bll.InstallVdsInternalCommand] (org.ovirt.thread.pool-6-thread-4) [461323e4] Lock Acquired to object EngineLock [exclusiveLocks= key: 14369bd0-907d-40d7-be1e-bd35af6d1488 value: VDS >, sharedLocks= ] >2014-09-09 15:37:08,931 INFO [org.ovirt.engine.core.bll.InstallVdsInternalCommand] (org.ovirt.thread.pool-6-thread-4) [461323e4] Running command: InstallVdsInternalCommand internal: true. Entities affected : ID: 14369bd0-907d-40d7-be1e-bd35af6d1488 Type: VDS >2014-09-09 15:37:08,938 INFO [org.ovirt.engine.core.bll.InstallVdsInternalCommand] (org.ovirt.thread.pool-6-thread-4) [461323e4] Before Installation host 14369bd0-907d-40d7-be1e-bd35af6d1488, rack >2014-09-09 15:37:08,956 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-6-thread-4) [461323e4] START, SetVdsStatusVDSCommand(HostName = rack, HostId = 14369bd0-907d-40d7-be1e-bd35af6d1488, status=Installing, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 47873fe0 >2014-09-09 15:37:09,002 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-6-thread-4) [461323e4] FINISH, SetVdsStatusVDSCommand, log id: 47873fe0 >2014-09-09 15:37:09,051 INFO [org.ovirt.engine.core.bll.InstallerMessages] (org.ovirt.thread.pool-6-thread-4) [461323e4] Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Connected to host dell-r210ii-16.rhev.lab.eng.brq.redhat.com with SSH key fingerprint: 3f:fe:39:f2:21:7b:9c:a0:53:49:a6:35:b0:6c:4a:ff >2014-09-09 15:37:09,102 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-4) [461323e4] Correlation ID: 461323e4, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Connected to host dell-r210ii-16.rhev.lab.eng.brq.redhat.com with SSH key fingerprint: 3f:fe:39:f2:21:7b:9c:a0:53:49:a6:35:b0:6c:4a:ff. >2014-09-09 15:37:09,123 INFO [org.ovirt.engine.core.bll.VdsDeploy] (org.ovirt.thread.pool-6-thread-4) [461323e4] Installation of dell-r210ii-16.rhev.lab.eng.brq.redhat.com. Executing command via SSH umask 0077; MYTMP="$(TMPDIR="${OVIRT_TMPDIR}" mktemp -t ovirt-XXXXXXXXXX)"; trap "chmod -R u+rwX \"${MYTMP}\" > /dev/null 2>&1; rm -fr \"${MYTMP}\" > /dev/null 2>&1" 0; rm -fr "${MYTMP}" && mkdir "${MYTMP}" && tar --warning=no-timestamp -C "${MYTMP}" -x && "${MYTMP}"/setup DIALOG/dialect=str:machine DIALOG/customization=bool:True < /var/cache/ovirt-engine/ovirt-host-deploy.tar >2014-09-09 15:37:09,123 INFO [org.ovirt.engine.core.utils.archivers.tar.CachedTar] (org.ovirt.thread.pool-6-thread-4) Tarball '/var/cache/ovirt-engine/ovirt-host-deploy.tar' is missing, creating >2014-09-09 15:37:09,367 INFO [org.ovirt.engine.core.uutils.ssh.SSHDialog] (org.ovirt.thread.pool-6-thread-4) SSH execute root@dell-r210ii-16.rhev.lab.eng.brq.redhat.com 'umask 0077; MYTMP="$(TMPDIR="${OVIRT_TMPDIR}" mktemp -t ovirt-XXXXXXXXXX)"; trap "chmod -R u+rwX \"${MYTMP}\" > /dev/null 2>&1; rm -fr \"${MYTMP}\" > /dev/null 2>&1" 0; rm -fr "${MYTMP}" && mkdir "${MYTMP}" && tar --warning=no-timestamp -C "${MYTMP}" -x && "${MYTMP}"/setup DIALOG/dialect=str:machine DIALOG/customization=bool:True' >2014-09-09 15:37:10,269 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Stage: Initializing >2014-09-09 15:37:10,287 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 461323e4, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Stage: Initializing. >2014-09-09 15:37:10,312 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Stage: Environment setup >2014-09-09 15:37:10,340 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 461323e4, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Stage: Environment setup. >2014-09-09 15:37:10,340 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Stage: Environment packages setup >2014-09-09 15:37:10,362 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 461323e4, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Stage: Environment packages setup. >2014-09-09 15:37:10,363 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Stage: Programs detection >2014-09-09 15:37:10,378 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 461323e4, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Stage: Programs detection. >2014-09-09 15:37:10,378 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Stage: Environment customization >2014-09-09 15:37:10,389 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 461323e4, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Stage: Environment customization. >2014-09-09 15:37:10,389 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Kdump unsupported >2014-09-09 15:37:10,400 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 461323e4, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Kdump unsupported. >2014-09-09 15:37:10,417 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Logs at host located at: '/tmp/ovirt-host-deploy-20140909153709-84s9io.log' >2014-09-09 15:37:10,433 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 461323e4, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Logs at host located at: '/tmp/ovirt-host-deploy-20140909153709-84s9io.log'. >2014-09-09 15:37:10,535 INFO [org.ovirt.engine.core.bll.VdsDeploy] (VdsDeploy) Host dell-r210ii-16.rhev.lab.eng.brq.redhat.com reports unique id 4C4C4544-0034-5310-8052-B4C04F4A354A >2014-09-09 15:37:10,538 INFO [org.ovirt.engine.core.bll.VdsDeploy] (VdsDeploy) Assigning unique id 4C4C4544-0034-5310-8052-B4C04F4A354A to Host dell-r210ii-16.rhev.lab.eng.brq.redhat.com >2014-09-09 15:37:10,658 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Disabling Kdump integration >2014-09-09 15:37:10,679 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 461323e4, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Disabling Kdump integration. >2014-09-09 15:37:10,700 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Stage: Setup validation >2014-09-09 15:37:10,722 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 461323e4, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Stage: Setup validation. >2014-09-09 15:37:10,747 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Hardware supports virtualization >2014-09-09 15:37:10,756 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 461323e4, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Hardware supports virtualization. >2014-09-09 15:37:10,757 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Stage: Transaction setup >2014-09-09 15:37:10,769 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 461323e4, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Stage: Transaction setup. >2014-09-09 15:37:10,769 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Stage: Misc configuration >2014-09-09 15:37:10,777 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 461323e4, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Stage: Misc configuration. >2014-09-09 15:37:10,777 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Stage: Package installation >2014-09-09 15:37:10,788 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 461323e4, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Stage: Package installation. >2014-09-09 15:37:11,756 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-100) Initializing Host: rack >2014-09-09 15:37:18,291 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Stage: Misc configuration >2014-09-09 15:37:18,304 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 461323e4, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Stage: Misc configuration. >2014-09-09 15:37:18,369 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Setting time >2014-09-09 15:37:18,387 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 461323e4, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Setting time. >2014-09-09 15:37:21,133 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Enrolling certificate >2014-09-09 15:37:21,179 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 461323e4, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Enrolling certificate. >2014-09-09 15:37:22,541 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Stage: Transaction commit >2014-09-09 15:37:22,556 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 461323e4, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Stage: Transaction commit. >2014-09-09 15:37:22,556 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Stage: Closing up >2014-09-09 15:37:22,581 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 461323e4, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Stage: Closing up. >2014-09-09 15:37:23,418 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Stopping libvirtd >2014-09-09 15:37:23,429 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 461323e4, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Stopping libvirtd. >2014-09-09 15:37:23,705 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Starting vdsm >2014-09-09 15:37:23,718 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 461323e4, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Starting vdsm. >2014-09-09 15:37:25,687 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Stage: Pre-termination >2014-09-09 15:37:25,710 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 461323e4, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Stage: Pre-termination. >2014-09-09 15:37:25,733 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Retrieving installation logs to: '/var/log/ovirt-engine/host-deploy/ovirt-20140909153725-dell-r210ii-16.rhev.lab.eng.brq.redhat.com-461323e4.log' >2014-09-09 15:37:25,745 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 461323e4, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Retrieving installation logs to: '/var/log/ovirt-engine/host-deploy/ovirt-20140909153725-dell-r210ii-16.rhev.lab.eng.brq.redhat.com-461323e4.log'. >2014-09-09 15:37:25,972 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Stage: Termination >2014-09-09 15:37:25,987 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 461323e4, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Stage: Termination. >2014-09-09 15:37:26,129 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to dell-r210ii-16.rhev.lab.eng.brq.redhat.com/10.34.62.208 >2014-09-09 15:37:26,476 INFO [org.ovirt.engine.core.bll.network.NetworkConfigurator] (org.ovirt.thread.pool-6-thread-4) [461323e4] Engine managed to communicate with VDSM agent on host rack >2014-09-09 15:37:27,099 WARN [org.ovirt.engine.core.bll.network.NetworkConfigurator] (org.ovirt.thread.pool-6-thread-4) [461323e4] The lastClientInterface ovirtmgmt of host rack is not a valid interface for the mangement network. If the interface is a bridge, it should be torn-down manually. >2014-09-09 15:37:27,105 ERROR [org.ovirt.engine.core.bll.InstallVdsInternalCommand] (org.ovirt.thread.pool-6-thread-4) [461323e4] Host installation failed for host 14369bd0-907d-40d7-be1e-bd35af6d1488, rack.: org.ovirt.engine.core.bll.VdsCommand$VdsInstallException: Failed to configure management network on the host > at org.ovirt.engine.core.bll.InstallVdsInternalCommand.configureManagementNetwork(InstallVdsInternalCommand.java:206) [bll.jar:] > at org.ovirt.engine.core.bll.InstallVdsInternalCommand.installHost(InstallVdsInternalCommand.java:164) [bll.jar:] > at org.ovirt.engine.core.bll.InstallVdsInternalCommand.executeCommand(InstallVdsInternalCommand.java:78) [bll.jar:] > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1167) [bll.jar:] > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1298) [bll.jar:] > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1920) [bll.jar:] > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) [utils.jar:] > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) [utils.jar:] > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1319) [bll.jar:] > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:344) [bll.jar:] > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:430) [bll.jar:] > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:411) [bll.jar:] > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:616) [bll.jar:] > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_65] > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptor.java:52) [jboss-as-ee.jar:7.4.0.Final-redhat-19] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:63) [jboss-as-ee.jar:7.4.0.Final-redhat-19] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:374) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.as.weld.ejb.Jsr299BindingsInterceptor.delegateInterception(Jsr299BindingsInterceptor.java:74) [jboss-as-weld.jar:7.4.0.Final-redhat-19] > at org.jboss.as.weld.ejb.Jsr299BindingsInterceptor.doMethodInterception(Jsr299BindingsInterceptor.java:84) [jboss-as-weld.jar:7.4.0.Final-redhat-19] > at org.jboss.as.weld.ejb.Jsr299BindingsInterceptor.processInvocation(Jsr299BindingsInterceptor.java:97) [jboss-as-weld.jar:7.4.0.Final-redhat-19] > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:63) [jboss-as-ee.jar:7.4.0.Final-redhat-19] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:63) [jboss-as-ee.jar:7.4.0.Final-redhat-19] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) [jboss-as-ejb3.jar:7.4.0.Final-redhat-19] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.as.weld.ejb.EjbRequestScopeActivationInterceptor.processInvocation(EjbRequestScopeActivationInterceptor.java:93) [jboss-as-weld.jar:7.4.0.Final-redhat-19] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) [jboss-as-ee.jar:7.4.0.Final-redhat-19] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) [jboss-as-ejb3.jar:7.4.0.Final-redhat-19] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) [jboss-as-ejb3.jar:7.4.0.Final-redhat-19] > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) [jboss-as-ejb3.jar:7.4.0.Final-redhat-19] > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) [jboss-as-ejb3.jar:7.4.0.Final-redhat-19] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) [jboss-as-ejb3.jar:7.4.0.Final-redhat-19] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) [jboss-as-ejb3.jar:7.4.0.Final-redhat-19] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) [jboss-as-ejb3.jar:7.4.0.Final-redhat-19] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) [jboss-as-ee.jar:7.4.0.Final-redhat-19] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) [jboss-as-ee.jar:7.4.0.Final-redhat-19] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:185) [jboss-as-ee.jar:7.4.0.Final-redhat-19] > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) [jboss-as-ee.jar:7.4.0.Final-redhat-19] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:73) [jboss-as-ee.jar:7.4.0.Final-redhat-19] > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view6.runInternalAction(Unknown Source) [bll.jar:] > at org.ovirt.engine.core.bll.CommandBase.runInternalAction(CommandBase.java:2249) [bll.jar:] > at org.ovirt.engine.core.bll.AddVdsCommand$3.run(AddVdsCommand.java:195) [bll.jar:] > at org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil$InternalWrapperRunnable.run(ThreadPoolUtil.java:90) [utils.jar:] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_65] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_65] > at java.lang.Thread.run(Thread.java:745) [rt.jar:1.7.0_65] >Caused by: org.ovirt.engine.core.bll.network.NetworkConfigurator$NetworkConfiguratorException: lastClientIface ovirtmgmt is not a valid interface for management network > at org.ovirt.engine.core.bll.network.NetworkConfigurator.findNicToSetupManagementNetwork(NetworkConfigurator.java:170) [bll.jar:] > at org.ovirt.engine.core.bll.network.NetworkConfigurator.createManagementNetworkIfRequired(NetworkConfigurator.java:74) [bll.jar:] > at org.ovirt.engine.core.bll.InstallVdsInternalCommand.configureManagementNetwork(InstallVdsInternalCommand.java:198) [bll.jar:] > ... 70 more > >2014-09-09 15:37:27,124 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-6-thread-4) [461323e4] START, SetVdsStatusVDSCommand(HostName = rack, HostId = 14369bd0-907d-40d7-be1e-bd35af6d1488, status=NonOperational, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 7a39ef2a >2014-09-09 15:37:27,148 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-6-thread-4) [461323e4] FINISH, SetVdsStatusVDSCommand, log id: 7a39ef2a >2014-09-09 15:37:27,204 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-4) [461323e4] Correlation ID: 461323e4, Job ID: a335eda2-f1e7-4c42-a7c8-1714f9c188bd, Call Stack: null, Custom Event ID: -1, Message: Host rack installation failed. Failed to configure management network on the host. >2014-09-09 15:37:27,214 INFO [org.ovirt.engine.core.bll.InstallVdsInternalCommand] (org.ovirt.thread.pool-6-thread-4) [461323e4] Lock freed to object EngineLock [exclusiveLocks= key: 14369bd0-907d-40d7-be1e-bd35af6d1488 value: VDS >, sharedLocks= ] >2014-09-09 15:40:00,108 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-71) Autorecovering 1 hosts >2014-09-09 15:40:00,109 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-71) Autorecovering hosts id: 14369bd0-907d-40d7-be1e-bd35af6d1488, name : rack >2014-09-09 15:40:00,121 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-71) [240b41f8] Lock Acquired to object EngineLock [exclusiveLocks= key: 14369bd0-907d-40d7-be1e-bd35af6d1488 value: VDS >, sharedLocks= ] >2014-09-09 15:40:00,133 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-71) [240b41f8] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 14369bd0-907d-40d7-be1e-bd35af6d1488 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2014-09-09 15:40:00,133 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-71) [240b41f8] Before acquiring lock in order to prevent monitoring for host rack from data-center Default >2014-09-09 15:40:00,133 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-71) [240b41f8] Lock acquired, from now a monitoring of host will be skipped for host rack from data-center Default >2014-09-09 15:40:00,152 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-71) [240b41f8] START, SetVdsStatusVDSCommand(HostName = rack, HostId = 14369bd0-907d-40d7-be1e-bd35af6d1488, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 120bd141 >2014-09-09 15:40:00,193 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-71) [240b41f8] FINISH, SetVdsStatusVDSCommand, log id: 120bd141 >2014-09-09 15:40:00,329 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-71) [240b41f8] Activate finished. Lock released. Monitoring can run now for host rack from data-center Default >2014-09-09 15:40:00,346 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-71) [240b41f8] Correlation ID: 240b41f8, Call Stack: null, Custom Event ID: -1, Message: Host rack was autorecovered. >2014-09-09 15:40:00,347 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-71) [240b41f8] Lock freed to object EngineLock [exclusiveLocks= key: 14369bd0-907d-40d7-be1e-bd35af6d1488 value: VDS >, sharedLocks= ] >2014-09-09 15:40:03,178 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-87) START, GetHardwareInfoVDSCommand(HostName = rack, HostId = 14369bd0-907d-40d7-be1e-bd35af6d1488, vds=Host[rack,14369bd0-907d-40d7-be1e-bd35af6d1488]), log id: 5100fff0 >2014-09-09 15:40:03,248 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-87) FINISH, GetHardwareInfoVDSCommand, log id: 5100fff0 >2014-09-09 15:40:03,384 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-87) [6349b998] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 14369bd0-907d-40d7-be1e-bd35af6d1488 Type: VDS >2014-09-09 15:40:03,387 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-87) [6349b998] START, SetVdsStatusVDSCommand(HostName = rack, HostId = 14369bd0-907d-40d7-be1e-bd35af6d1488, status=NonOperational, nonOperationalReason=NETWORK_UNREACHABLE, stopSpmFailureLogged=false), log id: 69771b18 >2014-09-09 15:40:03,419 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-87) [6349b998] FINISH, SetVdsStatusVDSCommand, log id: 69771b18 >2014-09-09 15:40:03,461 ERROR [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-87) [6349b998] Host rack is set to Non-Operational, it is missing the following networks: rhevm >2014-09-09 15:40:03,505 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-87) [6349b998] Correlation ID: 6349b998, Job ID: 25cb8a86-e0de-4ea3-8ad7-776f5ba47e04, Call Stack: null, Custom Event ID: -1, Message: Host rack does not comply with the cluster Default networks, the following networks are missing on host: 'rhevm' >2014-09-09 15:40:03,538 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-87) [741656c7] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 14369bd0-907d-40d7-be1e-bd35af6d1488 Type: VDS >2014-09-09 15:40:03,790 INFO [org.ovirt.engine.core.bll.UpdateVdsGroupCommand] (DefaultQuartzScheduler_Worker-87) [15828890] Running command: UpdateVdsGroupCommand internal: true. Entities affected : ID: 00000001-0001-0001-0001-000000000399 Type: VdsGroupsAction group EDIT_CLUSTER_CONFIGURATION with role type ADMIN >2014-09-09 15:40:03,860 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-87) [15828890] Correlation ID: 15828890, Call Stack: null, Custom Event ID: -1, Message: Host cluster Default was updated by system >2014-09-09 15:40:03,953 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-87) [15828890] START, GetHardwareInfoVDSCommand(HostName = rack, HostId = 14369bd0-907d-40d7-be1e-bd35af6d1488, vds=Host[rack,14369bd0-907d-40d7-be1e-bd35af6d1488]), log id: 428f21ef >2014-09-09 15:40:03,958 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-87) [15828890] FINISH, GetHardwareInfoVDSCommand, log id: 428f21ef >2014-09-09 15:40:04,065 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-87) [75e70b98] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 14369bd0-907d-40d7-be1e-bd35af6d1488 Type: VDS >2014-09-09 15:40:04,067 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-87) [75e70b98] START, SetVdsStatusVDSCommand(HostName = rack, HostId = 14369bd0-907d-40d7-be1e-bd35af6d1488, status=NonOperational, nonOperationalReason=NETWORK_UNREACHABLE, stopSpmFailureLogged=false), log id: c95f15f >2014-09-09 15:40:04,081 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-87) [75e70b98] FINISH, SetVdsStatusVDSCommand, log id: c95f15f >2014-09-09 15:40:04,128 ERROR [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-87) [75e70b98] Host rack is set to Non-Operational, it is missing the following networks: rhevm >2014-09-09 15:40:04,144 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-87) [66060f81] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 14369bd0-907d-40d7-be1e-bd35af6d1488 Type: VDS >2014-09-09 15:40:04,157 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-87) [66060f81] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host rack was set to NonOperational. >2014-09-09 15:40:04,217 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVdsCommand] (DefaultQuartzScheduler_Worker-87) [66060f81] START, FullListVdsCommand(HostName = rack, HostId = 14369bd0-907d-40d7-be1e-bd35af6d1488, vds=Host[rack,14369bd0-907d-40d7-be1e-bd35af6d1488], vmIds=[e8715843-38c5-4188-b00e-b9ceadce2714]), log id: f7bb3dd >2014-09-09 15:40:04,232 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVdsCommand] (DefaultQuartzScheduler_Worker-87) [66060f81] FINISH, FullListVdsCommand, return: [{guestFQDN=, acpiEnable=true, emulatedMachine=rhel6.5.0, spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,susbredir,ssmartcard, vmId=e8715843-38c5-4188-b00e-b9ceadce2714, memGuaranteedSize=6100, transparentHugePages=true, displaySecurePort=-1, spiceSslCipherSuite=DEFAULT, cpuType=Westmere, smp=1, pauseCode=NOERR, bootMenuEnable=false, numaTune={mode=preferred}, custom={}, vmType=kvm, memSize=6100, smpCoresPerSocket=1, vmName=341, nice=0, username=, exitMessage=User shut down from within the guest, pid=29513, copyPasteEnable=true, displayIp=10.34.62.208, displayPort=-1, smartcardEnable=false, clientIp=, exitCode=0, fileTransferEnable=true, nicModel=rtl8139,pv, keyboardLayout=en-us, kvmEnable=true, displayNetwork=ovirtmgmt, exitReason=7, status=Down, timeOffset=7200, maxVCpus=16, guestIPs=, devices=[Ljava.lang.Object;@73f10c54, display=qxl}], log id: f7bb3dd >2014-09-09 15:40:04,234 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-87) [66060f81] Importing VM 341 as external-341, as it is running on the on Host, but does not exist in the engine. >2014-09-09 15:40:04,639 INFO [org.ovirt.engine.core.bll.AddVmFromScratchCommand] (DefaultQuartzScheduler_Worker-87) [73ac8439] Lock Acquired to object EngineLock [exclusiveLocks= key: external-341 value: VM_NAME >, sharedLocks= ] >2014-09-09 15:40:04,774 INFO [org.ovirt.engine.core.bll.AddVmFromScratchCommand] (DefaultQuartzScheduler_Worker-87) [73ac8439] Running command: AddVmFromScratchCommand internal: true. Entities affected : ID: 00000001-0001-0001-0001-000000000399 Type: VdsGroupsAction group CREATE_VM with role type USER >2014-09-09 15:40:05,899 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-87) [73ac8439] Correlation ID: 73ac8439, Call Stack: null, Custom Event ID: -1, Message: VM external-341 was created by <UNKNOWN>. >2014-09-09 15:40:05,900 INFO [org.ovirt.engine.core.bll.AddVmFromScratchCommand] (DefaultQuartzScheduler_Worker-87) [73ac8439] Lock freed to object EngineLock [exclusiveLocks= key: external-341 value: VM_NAME >, sharedLocks= ] >2014-09-09 15:40:05,922 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-87) [151c998] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 14369bd0-907d-40d7-be1e-bd35af6d1488 Type: VDS >2014-09-09 15:40:05,929 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-87) [7adc4c5d] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 14369bd0-907d-40d7-be1e-bd35af6d1488 Type: VDS >2014-09-09 15:40:05,932 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-87) [7adc4c5d] Host 14369bd0-907d-40d7-be1e-bd35af6d1488 : rack is already in NonOperational status for reason NETWORK_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-09-09 15:41:16,318 INFO [org.ovirt.engine.core.bll.network.host.SetupNetworksCommand] (ajp-/127.0.0.1:8702-2) [6d52f989] Running command: SetupNetworksCommand internal: false. Entities affected : ID: 14369bd0-907d-40d7-be1e-bd35af6d1488 Type: VDSAction group CONFIGURE_HOST_NETWORK with role type ADMIN >2014-09-09 15:41:16,326 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetupNetworksVDSCommand] (ajp-/127.0.0.1:8702-2) [6d52f989] START, SetupNetworksVDSCommand(HostName = rack, HostId = 14369bd0-907d-40d7-be1e-bd35af6d1488, force=false, checkConnectivity=true, conectivityTimeout=120, > networks=[rhevm {id=00000000-0000-0000-0000-000000000009, description=Management Network, comment=null, subnet=null, gateway=null, type=null, vlanId=null, stp=false, dataCenterId=00000002-0002-0002-0002-0000000001ad, mtu=0, vmNetwork=true, cluster=NetworkCluster {id={clusterId=null, networkId=null}, status=OPERATIONAL, display=true, required=true, migration=true}, providedBy=null, label=null, qosId=null}], > bonds=[], > interfaces=[em2 {id=c694570f-fe20-4bc0-a065-b27ab1f9e659, vdsId=14369bd0-907d-40d7-be1e-bd35af6d1488, name=em2, macAddress=d4:ae:52:c7:0d:8c, networkName=null, bondName=null, bootProtocol=DHCP, address=, subnet=, gateway=null, mtu=1500, bridged=false, speed=0, type=0, networkImplementationDetails=null}, > em1 {id=235e3339-c91b-47fc-a126-36efad304f0a, vdsId=14369bd0-907d-40d7-be1e-bd35af6d1488, name=em1, macAddress=d4:ae:52:c7:0d:8b, networkName=rhevm, bondName=null, bootProtocol=DHCP, address=null, subnet=null, gateway=null, mtu=0, bridged=true, speed=1000, type=2, networkImplementationDetails=null}], > removedNetworks=[ovirtmgmt], > removedBonds=[]), log id: 19556e72 >2014-09-09 15:41:16,333 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetupNetworksVDSCommand] (ajp-/127.0.0.1:8702-2) [6d52f989] FINISH, SetupNetworksVDSCommand, log id: 19556e72 >2014-09-09 15:44:16,479 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SetupNetworksVDSCommand] (ajp-/127.0.0.1:8702-2) [6d52f989] org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues' >2014-09-09 15:44:16,486 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SetupNetworksVDSCommand] (ajp-/127.0.0.1:8702-2) [6d52f989] Command SetupNetworksVDSCommand(HostName = rack, HostId = 14369bd0-907d-40d7-be1e-bd35af6d1488, force=false, checkConnectivity=true, conectivityTimeout=120, > networks=[rhevm {id=00000000-0000-0000-0000-000000000009, description=Management Network, comment=null, subnet=null, gateway=null, type=null, vlanId=null, stp=false, dataCenterId=00000002-0002-0002-0002-0000000001ad, mtu=0, vmNetwork=true, cluster=NetworkCluster {id={clusterId=null, networkId=null}, status=OPERATIONAL, display=true, required=true, migration=true}, providedBy=null, label=null, qosId=null}], > bonds=[], > interfaces=[em2 {id=c694570f-fe20-4bc0-a065-b27ab1f9e659, vdsId=14369bd0-907d-40d7-be1e-bd35af6d1488, name=em2, macAddress=d4:ae:52:c7:0d:8c, networkName=null, bondName=null, bootProtocol=DHCP, address=, subnet=, gateway=null, mtu=1500, bridged=false, speed=0, type=0, networkImplementationDetails=null}, > em1 {id=235e3339-c91b-47fc-a126-36efad304f0a, vdsId=14369bd0-907d-40d7-be1e-bd35af6d1488, name=em1, macAddress=d4:ae:52:c7:0d:8b, networkName=rhevm, bondName=null, bootProtocol=DHCP, address=null, subnet=null, gateway=null, mtu=0, bridged=true, speed=1000, type=2, networkImplementationDetails=null}], > removedNetworks=[ovirtmgmt], > removedBonds=[]) execution failed. Exception: VDSNetworkException: VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues' >2014-09-09 15:44:16,487 ERROR [org.ovirt.engine.core.bll.network.host.SetupNetworksCommand] (ajp-/127.0.0.1:8702-2) [6d52f989] Command org.ovirt.engine.core.bll.network.host.SetupNetworksCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues' (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-09-09 15:45:00,021 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-51) Autorecovering 1 hosts >2014-09-09 15:45:00,021 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-51) Autorecovering hosts id: 14369bd0-907d-40d7-be1e-bd35af6d1488, name : rack >2014-09-09 15:45:00,027 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-51) [4d71ac53] Lock Acquired to object EngineLock [exclusiveLocks= key: 14369bd0-907d-40d7-be1e-bd35af6d1488 value: VDS >, sharedLocks= ] >2014-09-09 15:45:00,028 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-51) [4d71ac53] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 14369bd0-907d-40d7-be1e-bd35af6d1488 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2014-09-09 15:45:00,028 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-51) [4d71ac53] Before acquiring lock in order to prevent monitoring for host rack from data-center Default >2014-09-09 15:45:00,028 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-51) [4d71ac53] Lock acquired, from now a monitoring of host will be skipped for host rack from data-center Default >2014-09-09 15:45:00,038 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-51) [4d71ac53] START, SetVdsStatusVDSCommand(HostName = rack, HostId = 14369bd0-907d-40d7-be1e-bd35af6d1488, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 32c6f377 >2014-09-09 15:45:00,061 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-51) [4d71ac53] FINISH, SetVdsStatusVDSCommand, log id: 32c6f377 >2014-09-09 15:45:00,113 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-51) [4d71ac53] Activate finished. Lock released. Monitoring can run now for host rack from data-center Default >2014-09-09 15:45:00,119 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-51) [4d71ac53] Lock freed to object EngineLock [exclusiveLocks= key: 14369bd0-907d-40d7-be1e-bd35af6d1488 value: VDS >, sharedLocks= ] >2014-09-09 15:45:00,622 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to dell-r210ii-16.rhev.lab.eng.brq.redhat.com/10.34.62.208 >2014-09-09 15:45:00,624 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.Reactor] (SSL Stomp Reactor) Unable to process messages: java.nio.channels.ClosedChannelException > at sun.nio.ch.SocketChannelImpl.ensureReadOpen(SocketChannelImpl.java:252) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:295) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.SSLEngineNioHelper.read(SSLEngineNioHelper.java:51) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.SSLClient.read(SSLClient.java:81) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.readBuffer(ReactorClient.java:203) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.stomp.StompCommonClient.processIncoming(StompCommonClient.java:90) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.process(ReactorClient.java:144) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.SSLClient.process(SSLClient.java:115) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.processChannels(Reactor.java:86) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:62) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 15:45:27,183 WARN [org.ovirt.engine.core.bll.network.host.SetupNetworksCommand] (ajp-/127.0.0.1:8702-2) [66268694] CanDoAction of action SetupNetworks failed. Reasons:VAR__ACTION__SETUP,VAR__TYPE__NETWORKS,VAR__HOST_STATUS__UP_MAINTENANCE_OR_NON_OPERATIONAL,ACTION_TYPE_FAILED_VDS_STATUS_ILLEGAL >2014-09-09 15:46:16,176 INFO [org.ovirt.engine.core.bll.storage.FenceVdsManualyCommand] (ajp-/127.0.0.1:8702-13) [7cdc5df5] Lock Acquired to object EngineLock [exclusiveLocks= key: 14369bd0-907d-40d7-be1e-bd35af6d1488 value: VDS_FENCE >, sharedLocks= ] >2014-09-09 15:46:16,177 WARN [org.ovirt.engine.core.bll.storage.FenceVdsManualyCommand] (ajp-/127.0.0.1:8702-13) [7cdc5df5] CanDoAction of action FenceVdsManualy failed. Reasons:VAR__ACTION__MANUAL_FENCE,VAR__TYPE__HOST,ACTION_TYPE_FAILED_VDS_NOT_MATCH_VALID_STATUS >2014-09-09 15:46:16,177 INFO [org.ovirt.engine.core.bll.storage.FenceVdsManualyCommand] (ajp-/127.0.0.1:8702-13) [7cdc5df5] Lock freed to object EngineLock [exclusiveLocks= key: 14369bd0-907d-40d7-be1e-bd35af6d1488 value: VDS_FENCE >, sharedLocks= ] >2014-09-09 15:47:41,805 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Loaded file "/usr/share/ovirt-engine/services/ovirt-engine/ovirt-engine.conf". >2014-09-09 15:47:41,806 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) The file "/etc/ovirt-engine/engine.conf" doesn't exist or isn't readable. Will return an empty set of properties. >2014-09-09 15:47:41,806 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Loaded file "/etc/ovirt-engine/engine.conf.d/10-rhevm-doc.conf". >2014-09-09 15:47:41,806 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Loaded file "/etc/ovirt-engine/engine.conf.d/10-setup-database.conf". >2014-09-09 15:47:41,806 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Loaded file "/etc/ovirt-engine/engine.conf.d/10-setup-jboss.conf". >2014-09-09 15:47:41,807 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Loaded file "/etc/ovirt-engine/engine.conf.d/10-setup-pki.conf". >2014-09-09 15:47:41,807 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Loaded file "/etc/ovirt-engine/engine.conf.d/10-setup-protocols.conf". >2014-09-09 15:47:41,807 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Loaded file "/etc/ovirt-engine/engine.conf.d/20-branding-rhev.conf". >2014-09-09 15:47:41,807 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Loaded file "/etc/ovirt-engine/engine.conf.d/50-redhat-support-plugin.conf". >2014-09-09 15:47:41,808 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "ENGINE_AJP_ENABLED" is "true". >2014-09-09 15:47:41,808 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "ENGINE_AJP_PORT" is "8702". >2014-09-09 15:47:41,808 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "ENGINE_APPS" is "engine.ear restapi.war legacy_restapi.war "/usr/share/ovirt-engine/branding/rhev-2.brand/applications/rhev.ear" redhat-support-plugin-rhev.ear". >2014-09-09 15:47:41,808 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "ENGINE_CACHE" is "/var/cache/ovirt-engine". >2014-09-09 15:47:41,818 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "ENGINE_DB_CHECK_INTERVAL" is "1000". >2014-09-09 15:47:41,818 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "ENGINE_DB_CONNECTION_TIMEOUT" is "300000". >2014-09-09 15:47:41,818 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "ENGINE_DB_DATABASE" is "engine". >2014-09-09 15:47:41,818 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "ENGINE_DB_DRIVER" is "org.postgresql.Driver". >2014-09-09 15:47:41,818 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "ENGINE_DB_HOST" is "localhost". >2014-09-09 15:47:41,818 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "ENGINE_DB_MAX_CONNECTIONS" is "100". >2014-09-09 15:47:41,818 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "ENGINE_DB_MIN_CONNECTIONS" is "1". >2014-09-09 15:47:41,818 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "ENGINE_DB_PASSWORD" is "***". >2014-09-09 15:47:41,819 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "ENGINE_DB_PORT" is "5432". >2014-09-09 15:47:41,819 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "ENGINE_DB_SECURED" is "False". >2014-09-09 15:47:41,819 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "ENGINE_DB_SECURED_VALIDATION" is "False". >2014-09-09 15:47:41,819 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "ENGINE_DB_URL" is "jdbc:postgresql://localhost:5432/engine?sslfactory=org.postgresql.ssl.NonValidatingFactory". >2014-09-09 15:47:41,819 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "ENGINE_DB_USER" is "engine". >2014-09-09 15:47:41,819 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "ENGINE_DEBUG_ADDRESS" is "". >2014-09-09 15:47:41,819 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "ENGINE_DOC" is "/usr/share/doc/ovirt-engine". >2014-09-09 15:47:41,819 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "ENGINE_ETC" is "/etc/ovirt-engine". >2014-09-09 15:47:41,836 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "ENGINE_EXTENSION_PATH" is "/usr/share/ovirt-engine/extensions.d:/etc/ovirt-engine/extensions.d". >2014-09-09 15:47:41,836 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "ENGINE_FQDN" is "pb-rh35.rhev.lab.eng.brq.redhat.com". >2014-09-09 15:47:41,836 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "ENGINE_GROUP" is "ovirt". >2014-09-09 15:47:41,836 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "ENGINE_HEAP_MAX" is "1g". >2014-09-09 15:47:41,837 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "ENGINE_HEAP_MIN" is "1g". >2014-09-09 15:47:41,837 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "ENGINE_HTTPS_ENABLED" is "false". >2014-09-09 15:47:41,837 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "ENGINE_HTTPS_PORT" is "None". >2014-09-09 15:47:41,837 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "ENGINE_HTTPS_PROTOCOLS" is "SSLv3,TLSv1,TLSv1.1,TLSv1.2". >2014-09-09 15:47:41,837 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "ENGINE_HTTP_ENABLED" is "false". >2014-09-09 15:47:41,837 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "ENGINE_HTTP_PORT" is "None". >2014-09-09 15:47:41,837 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "ENGINE_JAVA_MODULEPATH" is "/usr/share/ovirt-engine/modules". >2014-09-09 15:47:41,837 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "ENGINE_JVM_ARGS" is " -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath="/var/log/ovirt-engine/dump"". >2014-09-09 15:47:41,837 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "ENGINE_LOG" is "/var/log/ovirt-engine". >2014-09-09 15:47:41,838 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "ENGINE_LOG_TO_CONSOLE" is "false". >2014-09-09 15:47:41,838 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "ENGINE_MANUAL" is "/usr/share/doc/rhevm-doc/manual". >2014-09-09 15:47:41,838 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "ENGINE_PERM_MAX" is "256m". >2014-09-09 15:47:41,838 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "ENGINE_PERM_MIN" is "256m". >2014-09-09 15:47:41,838 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "ENGINE_PKI" is "/etc/pki/ovirt-engine". >2014-09-09 15:47:41,838 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "ENGINE_PKI_CA" is "/etc/pki/ovirt-engine/ca.pem". >2014-09-09 15:47:41,838 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "ENGINE_PKI_ENGINE_CERT" is "/etc/pki/ovirt-engine/certs/engine.cer". >2014-09-09 15:47:41,838 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "ENGINE_PKI_ENGINE_STORE" is "/etc/pki/ovirt-engine/keys/engine.p12". >2014-09-09 15:47:41,838 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "ENGINE_PKI_ENGINE_STORE_ALIAS" is "1". >2014-09-09 15:47:41,839 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "ENGINE_PKI_ENGINE_STORE_PASSWORD" is "***". >2014-09-09 15:47:41,839 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "ENGINE_PKI_TRUST_STORE" is "/etc/pki/ovirt-engine/.truststore". >2014-09-09 15:47:41,839 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "ENGINE_PKI_TRUST_STORE_PASSWORD" is "***". >2014-09-09 15:47:41,839 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "ENGINE_PROPERTIES" is " jsse.enableSNIExtension=false "java.security.krb5.conf=/etc/ovirt-engine/krb5.conf"". >2014-09-09 15:47:41,839 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "ENGINE_PROXY_ENABLED" is "true". >2014-09-09 15:47:41,839 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "ENGINE_PROXY_HTTPS_PORT" is "443". >2014-09-09 15:47:41,839 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "ENGINE_PROXY_HTTP_PORT" is "80". >2014-09-09 15:47:41,839 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "ENGINE_REPORTS_UI" is "/var/lib/ovirt-engine/reports.xml". >2014-09-09 15:47:41,840 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "ENGINE_STOP_INTERVAL" is "1". >2014-09-09 15:47:41,840 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "ENGINE_STOP_TIME" is "10". >2014-09-09 15:47:41,840 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "ENGINE_TMP" is "/var/tmp/ovirt-engine". >2014-09-09 15:47:41,840 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "ENGINE_UP_MARK" is "/var/lib/ovirt-engine/engine.up". >2014-09-09 15:47:41,840 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "ENGINE_URI" is "/ovirt-engine". >2014-09-09 15:47:41,840 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "ENGINE_USER" is "ovirt". >2014-09-09 15:47:41,840 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "ENGINE_USR" is "/usr/share/ovirt-engine". >2014-09-09 15:47:41,840 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "ENGINE_VAR" is "/var/lib/ovirt-engine". >2014-09-09 15:47:41,840 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "ENGINE_VERBOSE_GC" is "false". >2014-09-09 15:47:41,840 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "JBOSS_HOME" is "/usr/share/jbossas". >2014-09-09 15:47:41,841 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "JBOSS_RUNTIME" is "/var/lib/ovirt-engine/jboss_runtime". >2014-09-09 15:47:41,841 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "SENSITIVE_KEYS" is ",ENGINE_DB_PASSWORD,ENGINE_PKI_TRUST_STORE_PASSWORD,ENGINE_PKI_ENGINE_STORE_PASSWORD". >2014-09-09 15:47:50,521 INFO [org.ovirt.engine.core.bll.Backend] (ServerService Thread Pool -- 42) Start initializing Backend >2014-09-09 15:47:51,049 INFO [org.ovirt.engine.core.utils.osinfo.OsInfoPreferencesLoader] (ServerService Thread Pool -- 42) Loaded file /etc/ovirt-engine/osinfo.conf.d/00-defaults.properties >2014-09-09 15:47:51,230 INFO [org.ovirt.engine.core.bll.Backend] (ServerService Thread Pool -- 42) Running ovirt-engine 3.5.0-0.10.master.el6ev >2014-09-09 15:47:51,231 INFO [org.ovirt.engine.core.bll.CpuFlagsManagerHandler] (ServerService Thread Pool -- 42) Start initializing dictionaries >2014-09-09 15:47:51,234 INFO [org.ovirt.engine.core.bll.CpuFlagsManagerHandler] (ServerService Thread Pool -- 42) Finished initializing dictionaries >2014-09-09 15:47:51,235 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (ServerService Thread Pool -- 42) Start initializing AuditLogCleanupManager >2014-09-09 15:47:51,235 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (ServerService Thread Pool -- 42) Setting audit cleanup manager to run at: 35 35 3 * * ? >2014-09-09 15:47:51,277 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (ServerService Thread Pool -- 42) Finished initializing AuditLogCleanupManager >2014-09-09 15:47:51,277 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (ServerService Thread Pool -- 42) Start initializing CommandEntityCleanupManager >2014-09-09 15:47:51,278 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (ServerService Thread Pool -- 42) Setting command entity cleanup manager to run at: 35 35 3 * * ? >2014-09-09 15:47:51,280 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (ServerService Thread Pool -- 42) Finished initializing CommandEntityCleanupManager >2014-09-09 15:47:51,283 INFO [org.ovirt.engine.core.bll.TagsDirector] (ServerService Thread Pool -- 42) Start initializing TagsDirector >2014-09-09 15:47:51,309 INFO [org.ovirt.engine.core.bll.TagsDirector] (ServerService Thread Pool -- 42) Tag root added to tree >2014-09-09 15:47:51,337 INFO [org.ovirt.engine.core.bll.TagsDirector] (ServerService Thread Pool -- 42) Finished initializing TagsDirector >2014-09-09 15:47:51,337 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (ServerService Thread Pool -- 42) Start initializing IsoDomainListSyncronizer >2014-09-09 15:47:51,353 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (ServerService Thread Pool -- 42) Finished initializing IsoDomainListSyncronizer >2014-09-09 15:47:51,432 INFO [org.ovirt.engine.core.bll.Backend] (ServerService Thread Pool -- 42) Completed initializing handlers >2014-09-09 15:47:51,475 INFO [org.ovirt.engine.core.utils.ErrorTranslatorImpl] (ServerService Thread Pool -- 42) Start initializing ErrorTranslatorImpl >2014-09-09 15:47:51,498 WARN [org.ovirt.engine.core.utils.ErrorTranslatorImpl] (ServerService Thread Pool -- 42) Code MAC_ADDRESS_IS_IN_USE appears more than once in string table. >2014-09-09 15:47:51,500 INFO [org.ovirt.engine.core.utils.ErrorTranslatorImpl] (ServerService Thread Pool -- 42) Finished initializing ErrorTranslatorImpl >2014-09-09 15:47:51,500 INFO [org.ovirt.engine.core.utils.ErrorTranslatorImpl] (ServerService Thread Pool -- 42) Start initializing ErrorTranslatorImpl >2014-09-09 15:47:51,500 INFO [org.ovirt.engine.core.utils.ErrorTranslatorImpl] (ServerService Thread Pool -- 42) Finished initializing ErrorTranslatorImpl >2014-09-09 15:47:51,500 INFO [org.ovirt.engine.core.bll.Backend] (ServerService Thread Pool -- 42) Mark incomplete jobs as UNKNOWN >2014-09-09 15:47:51,561 INFO [org.ovirt.engine.core.bll.job.JobRepositoryCleanupManager] (ServerService Thread Pool -- 42) Start initializing JobRepositoryCleanupManager >2014-09-09 15:47:51,563 INFO [org.ovirt.engine.core.bll.job.JobRepositoryCleanupManager] (ServerService Thread Pool -- 42) Finished initializing JobRepositoryCleanupManager >2014-09-09 15:47:51,563 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (ServerService Thread Pool -- 42) Start initializing AutoRecoveryManager >2014-09-09 15:47:51,565 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (ServerService Thread Pool -- 42) Finished initializing AutoRecoveryManager >2014-09-09 15:47:51,567 INFO [org.ovirt.engine.core.dal.job.ExecutionMessageDirector] (ServerService Thread Pool -- 42) Start initializing ExecutionMessageDirector >2014-09-09 15:47:51,576 INFO [org.ovirt.engine.core.dal.job.ExecutionMessageDirector] (ServerService Thread Pool -- 42) Finished initializing ExecutionMessageDirector >2014-09-09 15:47:51,754 INFO [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (ServerService Thread Pool -- 42) Loading extension 'builtin-authn-internal' >2014-09-09 15:47:51,764 INFO [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (ServerService Thread Pool -- 42) Extension 'builtin-authn-internal' loaded >2014-09-09 15:47:51,766 INFO [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (ServerService Thread Pool -- 42) Loading extension 'internal' >2014-09-09 15:47:51,768 INFO [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (ServerService Thread Pool -- 42) Extension 'internal' loaded >2014-09-09 15:47:51,775 INFO [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (ServerService Thread Pool -- 42) Initializing extension 'builtin-authn-internal' >2014-09-09 15:47:51,775 INFO [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (ServerService Thread Pool -- 42) Extension 'builtin-authn-internal' initialized >2014-09-09 15:47:51,776 INFO [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (ServerService Thread Pool -- 42) Initializing extension 'internal' >2014-09-09 15:47:51,776 INFO [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (ServerService Thread Pool -- 42) Extension 'internal' initialized >2014-09-09 15:47:51,776 INFO [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (ServerService Thread Pool -- 42) Start of enabled extensions list >2014-09-09 15:47:51,776 INFO [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (ServerService Thread Pool -- 42) Instance name: 'builtin-authn-internal', Extension name: 'Internal Authn (Built-in)', Version: 'N/A', Notes: '', License: 'ASL 2.0', Home: 'http://www.ovirt.org', Author 'The oVirt Project', Build interface Version: '0', File: 'N/A', Initialized: 'true' >2014-09-09 15:47:51,776 INFO [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (ServerService Thread Pool -- 42) Instance name: 'internal', Extension name: 'Internal Authz (Built-in)', Version: 'N/A', Notes: '', License: 'ASL 2.0', Home: 'http://www.ovirt.org', Author 'The oVirt Project', Build interface Version: '0', File: 'N/A', Initialized: 'true' >2014-09-09 15:47:51,776 INFO [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (ServerService Thread Pool -- 42) End of enabled extensions list >2014-09-09 15:47:51,982 INFO [org.ovirt.engine.core.bll.aaa.DbUserCacheManager] (ServerService Thread Pool -- 47) Start initializing DbUserCacheManager >2014-09-09 15:47:51,991 INFO [org.ovirt.engine.core.bll.aaa.DbUserCacheManager] (ServerService Thread Pool -- 47) Finished initializing DbUserCacheManager >2014-09-09 15:47:52,037 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (ServerService Thread Pool -- 47) Initialization of AsyncTaskManager completed successfully. >2014-09-09 15:47:52,039 INFO [org.ovirt.engine.core.vdsbroker.ResourceManager] (ServerService Thread Pool -- 47) Start initializing ResourceManager >2014-09-09 15:47:52,204 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-1) Principal admin::internal synchronized >2014-09-09 15:47:52,313 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (ServerService Thread Pool -- 47) Entered VdsManager constructor >2014-09-09 15:47:52,383 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (ServerService Thread Pool -- 47) Initialize vdsBroker (dell-r210ii-16.rhev.lab.eng.brq.redhat.com,54,321) >2014-09-09 15:47:52,544 INFO [org.ovirt.engine.core.vdsbroker.ResourceManager] (ServerService Thread Pool -- 47) VDS 14369bd0-907d-40d7-be1e-bd35af6d1488 was added to the Resource Manager >2014-09-09 15:47:52,580 INFO [org.ovirt.engine.core.vdsbroker.ResourceManager] (ServerService Thread Pool -- 47) Finished initializing ResourceManager >2014-09-09 15:47:52,586 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (ServerService Thread Pool -- 47) Initialization of OvfDataUpdater completed successfully. >2014-09-09 15:47:52,587 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (ServerService Thread Pool -- 47) Start scheduling to enable vds load balancer >2014-09-09 15:47:52,588 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (ServerService Thread Pool -- 47) Finished scheduling to enable vds load balancer >2014-09-09 15:47:52,588 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (ServerService Thread Pool -- 47) Start HA Reservation check >2014-09-09 15:47:52,589 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (ServerService Thread Pool -- 47) Finished HA Reservation check >2014-09-09 15:47:52,602 INFO [org.ovirt.engine.core.bll.network.MacPoolManagerRanges] (org.ovirt.thread.pool-6-thread-1) Start initializing MacPoolManagerRanges >2014-09-09 15:47:52,610 INFO [org.ovirt.engine.core.bll.InitBackendServicesOnStartupBean] (ServerService Thread Pool -- 47) Init VM custom properties utilities >2014-09-09 15:47:52,612 INFO [org.ovirt.engine.core.bll.InitBackendServicesOnStartupBean] (ServerService Thread Pool -- 47) Init device custom properties utilities >2014-09-09 15:47:52,617 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (ServerService Thread Pool -- 47) Initializing Scheduling manager >2014-09-09 15:47:52,656 INFO [org.ovirt.engine.core.bll.network.MacPoolManagerRanges] (org.ovirt.thread.pool-6-thread-1) Finished initializing. Available MACs in pool: 65 >2014-09-09 15:47:52,689 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (ServerService Thread Pool -- 47) External scheduler disabled, discovery skipped >2014-09-09 15:47:52,689 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (ServerService Thread Pool -- 47) Initialized Scheduling manager >2014-09-09 15:47:52,689 INFO [org.ovirt.engine.core.bll.dwh.DwhHeartBeat] (ServerService Thread Pool -- 47) Initializing DWH Heart Beat >2014-09-09 15:47:52,690 INFO [org.ovirt.engine.core.bll.dwh.DwhHeartBeat] (ServerService Thread Pool -- 47) DWH Heart Beat initialized >2014-09-09 15:47:54,584 INFO [org.ovirt.engine.ui.frontend.server.gwt.plugin.PluginDataManager] (ajp-/127.0.0.1:8702-1) Reading UI plugin descriptor [/usr/share/ovirt-engine/ui-plugins/redhat_support_plugin_rhev.json] >2014-09-09 15:47:54,601 INFO [org.ovirt.engine.ui.frontend.server.gwt.plugin.PluginDataManager] (ajp-/127.0.0.1:8702-1) Reading UI plugin configuration [/etc/ovirt-engine/ui-plugins/redhat_support_plugin_rhev-config.json] >2014-09-09 15:47:55,079 INFO [org.ovirt.engine.core.utils.servlet.UnsupportedLocaleHelper] (ajp-/127.0.0.1:8702-1) Invalid locale found in configuration: >2014-09-09 15:47:55,079 INFO [org.ovirt.engine.core.utils.servlet.UnsupportedLocaleHelper] (ajp-/127.0.0.1:8702-1) Invalid locale found in configuration: >2014-09-09 15:47:55,794 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to dell-r210ii-16.rhev.lab.eng.brq.redhat.com/10.34.62.208 >2014-09-09 15:47:56,565 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-6) START, GetHardwareInfoVDSCommand(HostName = rack, HostId = 14369bd0-907d-40d7-be1e-bd35af6d1488, vds=Host[rack,14369bd0-907d-40d7-be1e-bd35af6d1488]), log id: 638aa1c2 >2014-09-09 15:47:56,578 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-6) FINISH, GetHardwareInfoVDSCommand, log id: 638aa1c2 >2014-09-09 15:47:56,834 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-6) [1da041f2] AuditLogType: UNASSIGNED not exist in string table >2014-09-09 15:47:56,835 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-6) [1da041f2] AuditLogType: VDS_AUTO_FENCE_STATUS not exist in string table >2014-09-09 15:47:56,835 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-6) [1da041f2] AuditLogType: VDS_AUTO_FENCE_STATUS_FAILED not exist in string table >2014-09-09 15:47:56,835 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-6) [1da041f2] AuditLogType: VDS_AUTO_FENCE_FAILED_CALL_FENCE_SPM not exist in string table >2014-09-09 15:47:56,835 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-6) [1da041f2] AuditLogType: VDS_HIGH_NETWORK_USE not exist in string table >2014-09-09 15:47:56,835 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-6) [1da041f2] AuditLogType: USER_FAILED_REMOVE_VM not exist in string table >2014-09-09 15:47:56,835 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-6) [1da041f2] AuditLogType: USER_RUN_UNLOCK_ENTITY_SCRIPT not exist in string table >2014-09-09 15:47:56,836 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-6) [1da041f2] AuditLogType: VDS_NETWORK_MTU_DIFFER_FROM_LOGICAL_NETWORK not exist in string table >2014-09-09 15:47:56,836 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-6) [1da041f2] AuditLogType: STORAGE_ACTIVATE_ASYNC not exist in string table >2014-09-09 15:47:56,836 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-6) [1da041f2] AuditLogType: DWH_STOPPED not exist in string table >2014-09-09 15:47:56,836 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-6) [1da041f2] AuditLogType: DWH_STARTED not exist in string table >2014-09-09 15:47:56,836 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-6) [1da041f2] AuditLogType: DWH_ERROR not exist in string table >2014-09-09 15:47:56,836 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-6) [1da041f2] AuditLogType: USER_ADDED_DISK_PROFILE not exist in string table >2014-09-09 15:47:56,836 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-6) [1da041f2] AuditLogType: USER_FAILED_TO_ADD_DISK_PROFILE not exist in string table >2014-09-09 15:47:56,836 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-6) [1da041f2] AuditLogType: USER_REMOVED_DISK_PROFILE not exist in string table >2014-09-09 15:47:56,837 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-6) [1da041f2] AuditLogType: USER_FAILED_TO_REMOVE_DISK_PROFILE not exist in string table >2014-09-09 15:47:56,837 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-6) [1da041f2] AuditLogType: USER_UPDATED_DISK_PROFILE not exist in string table >2014-09-09 15:47:56,837 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-6) [1da041f2] AuditLogType: USER_FAILED_TO_UPDATE_DISK_PROFILE not exist in string table >2014-09-09 15:47:56,837 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-6) [1da041f2] AuditLogType: USER_ADDED_CPU_PROFILE not exist in string table >2014-09-09 15:47:56,837 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-6) [1da041f2] AuditLogType: USER_FAILED_TO_ADD_CPU_PROFILE not exist in string table >2014-09-09 15:47:56,837 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-6) [1da041f2] AuditLogType: USER_REMOVED_CPU_PROFILE not exist in string table >2014-09-09 15:47:56,837 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-6) [1da041f2] AuditLogType: USER_FAILED_TO_REMOVE_CPU_PROFILE not exist in string table >2014-09-09 15:47:56,837 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-6) [1da041f2] AuditLogType: USER_UPDATED_CPU_PROFILE not exist in string table >2014-09-09 15:47:56,837 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-6) [1da041f2] AuditLogType: USER_FAILED_TO_UPDATE_CPU_PROFILE not exist in string table >2014-09-09 15:47:56,837 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-6) [1da041f2] AuditLogType: USER_REMOVE_AUDIT_LOG not exist in string table >2014-09-09 15:47:56,838 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-6) [1da041f2] AuditLogType: USER_REMOVE_AUDIT_LOG_FAILED not exist in string table >2014-09-09 15:47:56,838 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-6) [1da041f2] AuditLogType: USER_CLEAR_ALL_DISMISSED_AUDIT_LOG not exist in string table >2014-09-09 15:47:56,838 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-6) [1da041f2] AuditLogType: USER_CLEAR_ALL_DISMISSED_AUDIT_LOG_FAILED not exist in string table >2014-09-09 15:47:56,977 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-6) [1da041f2] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 14369bd0-907d-40d7-be1e-bd35af6d1488 Type: VDS >2014-09-09 15:47:56,982 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-6) [1da041f2] START, SetVdsStatusVDSCommand(HostName = rack, HostId = 14369bd0-907d-40d7-be1e-bd35af6d1488, status=NonOperational, nonOperationalReason=NETWORK_UNREACHABLE, stopSpmFailureLogged=false), log id: 4ac5f9cf >2014-09-09 15:47:57,023 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-6) [1da041f2] FINISH, SetVdsStatusVDSCommand, log id: 4ac5f9cf >2014-09-09 15:47:57,059 ERROR [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-6) [1da041f2] Host rack is set to Non-Operational, it is missing the following networks: rhevm >2014-09-09 15:47:57,506 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-6) [1da041f2] Correlation ID: 1da041f2, Job ID: e0d83450-ef71-4dcb-bd0c-25855560fee0, Call Stack: null, Custom Event ID: -1, Message: Host rack does not comply with the cluster Default networks, the following networks are missing on host: 'rhevm' >2014-09-09 15:47:57,580 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-6) [390b0eea] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 14369bd0-907d-40d7-be1e-bd35af6d1488 Type: VDS >2014-09-09 15:47:57,720 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-6) [390b0eea] START, GetHardwareInfoVDSCommand(HostName = rack, HostId = 14369bd0-907d-40d7-be1e-bd35af6d1488, vds=Host[rack,14369bd0-907d-40d7-be1e-bd35af6d1488]), log id: 6bee3d2e >2014-09-09 15:47:57,727 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-6) [390b0eea] FINISH, GetHardwareInfoVDSCommand, log id: 6bee3d2e >2014-09-09 15:47:57,919 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-6) [702723b6] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 14369bd0-907d-40d7-be1e-bd35af6d1488 Type: VDS >2014-09-09 15:47:57,940 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-6) [702723b6] START, SetVdsStatusVDSCommand(HostName = rack, HostId = 14369bd0-907d-40d7-be1e-bd35af6d1488, status=NonOperational, nonOperationalReason=NETWORK_UNREACHABLE, stopSpmFailureLogged=false), log id: 22d82eef >2014-09-09 15:47:57,978 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-6) [702723b6] FINISH, SetVdsStatusVDSCommand, log id: 22d82eef >2014-09-09 15:47:58,001 ERROR [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-6) [702723b6] Host rack is set to Non-Operational, it is missing the following networks: rhevm >2014-09-09 15:47:58,040 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-6) [18fdd1da] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 14369bd0-907d-40d7-be1e-bd35af6d1488 Type: VDS >2014-09-09 15:47:58,075 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-6) [18fdd1da] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host rack was set to NonOperational. >2014-09-09 15:47:58,229 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (DefaultQuartzScheduler_Worker-6) [18fdd1da] START, DestroyVDSCommand(HostName = rack, HostId = 14369bd0-907d-40d7-be1e-bd35af6d1488, vmId=e8715843-38c5-4188-b00e-b9ceadce2714, force=false, secondsToWait=0, gracefully=false, reason=), log id: 42dbc279 >2014-09-09 15:47:58,348 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (DefaultQuartzScheduler_Worker-6) [18fdd1da] FINISH, DestroyVDSCommand, log id: 42dbc279 >2014-09-09 15:47:58,678 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-6) [18fdd1da] Initializing Host: rack >2014-09-09 15:47:58,761 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-6) [6357073f] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 14369bd0-907d-40d7-be1e-bd35af6d1488 Type: VDS >2014-09-09 15:47:58,765 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-6) [6357073f] Host 14369bd0-907d-40d7-be1e-bd35af6d1488 : rack is already in NonOperational status for reason NETWORK_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-09-09 15:48:05,047 INFO [org.ovirt.engine.core.bll.aaa.LoginAdminUserCommand] (ajp-/127.0.0.1:8702-3) Running command: LoginAdminUserCommand internal: false. >2014-09-09 15:48:05,087 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-3) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-09-09 15:48:09,021 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp-/127.0.0.1:8702-4) Running command: LoginUserCommand internal: false. >2014-09-09 15:50:00,106 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-81) Autorecovering 1 hosts >2014-09-09 15:50:00,106 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-81) Autorecovering hosts id: 14369bd0-907d-40d7-be1e-bd35af6d1488, name : rack >2014-09-09 15:50:00,119 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-81) [24b7b64b] Lock Acquired to object EngineLock [exclusiveLocks= key: 14369bd0-907d-40d7-be1e-bd35af6d1488 value: VDS >, sharedLocks= ] >2014-09-09 15:50:00,121 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-81) [24b7b64b] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 14369bd0-907d-40d7-be1e-bd35af6d1488 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2014-09-09 15:50:00,122 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-81) [24b7b64b] Before acquiring lock in order to prevent monitoring for host rack from data-center Default >2014-09-09 15:50:00,122 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-81) [24b7b64b] Lock acquired, from now a monitoring of host will be skipped for host rack from data-center Default >2014-09-09 15:50:00,142 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-81) [24b7b64b] START, SetVdsStatusVDSCommand(HostName = rack, HostId = 14369bd0-907d-40d7-be1e-bd35af6d1488, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 10d0f86a >2014-09-09 15:50:00,177 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-81) [24b7b64b] FINISH, SetVdsStatusVDSCommand, log id: 10d0f86a >2014-09-09 15:50:00,304 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-81) [24b7b64b] Activate finished. Lock released. Monitoring can run now for host rack from data-center Default >2014-09-09 15:50:00,336 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-81) [24b7b64b] Correlation ID: 24b7b64b, Call Stack: null, Custom Event ID: -1, Message: Host rack was autorecovered. >2014-09-09 15:50:00,336 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-81) [24b7b64b] Lock freed to object EngineLock [exclusiveLocks= key: 14369bd0-907d-40d7-be1e-bd35af6d1488 value: VDS >, sharedLocks= ] >2014-09-09 15:50:01,899 WARN [org.ovirt.engine.core.bll.MaintenanceNumberOfVdssCommand] (ajp-/127.0.0.1:8702-8) [6bb4a5f] CanDoAction of action MaintenanceNumberOfVdss failed. Reasons:VAR__TYPE__HOST,VAR__ACTION__MAINTENANCE,VDS_CANNOT_MAINTENANCE_VDS_IS_NOT_OPERATIONAL >2014-09-09 15:50:02,202 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-87) START, GetHardwareInfoVDSCommand(HostName = rack, HostId = 14369bd0-907d-40d7-be1e-bd35af6d1488, vds=Host[rack,14369bd0-907d-40d7-be1e-bd35af6d1488]), log id: 4c4b22dd >2014-09-09 15:50:02,209 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-87) FINISH, GetHardwareInfoVDSCommand, log id: 4c4b22dd >2014-09-09 15:50:02,414 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-87) [46193dd4] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 14369bd0-907d-40d7-be1e-bd35af6d1488 Type: VDS >2014-09-09 15:50:02,446 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-87) [46193dd4] START, SetVdsStatusVDSCommand(HostName = rack, HostId = 14369bd0-907d-40d7-be1e-bd35af6d1488, status=NonOperational, nonOperationalReason=NETWORK_UNREACHABLE, stopSpmFailureLogged=false), log id: 29c2bda5 >2014-09-09 15:50:02,479 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-87) [46193dd4] FINISH, SetVdsStatusVDSCommand, log id: 29c2bda5 >2014-09-09 15:50:02,488 ERROR [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-87) [46193dd4] Host rack is set to Non-Operational, it is missing the following networks: rhevm >2014-09-09 15:50:02,517 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-87) [46193dd4] Correlation ID: 46193dd4, Job ID: 17174584-81e5-4c6e-8f8f-41f6890fd205, Call Stack: null, Custom Event ID: -1, Message: Host rack does not comply with the cluster Default networks, the following networks are missing on host: 'rhevm' >2014-09-09 15:50:02,551 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-87) [a90f6d5] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 14369bd0-907d-40d7-be1e-bd35af6d1488 Type: VDS >2014-09-09 15:50:02,573 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-87) [a90f6d5] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host rack was set to NonOperational. >2014-09-09 15:50:02,634 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-87) [53e7c8ba] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 14369bd0-907d-40d7-be1e-bd35af6d1488 Type: VDS >2014-09-09 15:50:02,638 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-87) [53e7c8ba] Host 14369bd0-907d-40d7-be1e-bd35af6d1488 : rack is already in NonOperational status for reason NETWORK_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-09-09 15:50:20,919 INFO [org.ovirt.engine.core.bll.MaintenanceNumberOfVdssCommand] (org.ovirt.thread.pool-6-thread-10) [6a92276b] Running command: MaintenanceNumberOfVdssCommand internal: false. Entities affected : ID: 14369bd0-907d-40d7-be1e-bd35af6d1488 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2014-09-09 15:50:20,923 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-6-thread-10) [6a92276b] START, SetVdsStatusVDSCommand(HostName = rack, HostId = 14369bd0-907d-40d7-be1e-bd35af6d1488, status=PreparingForMaintenance, nonOperationalReason=NONE, stopSpmFailureLogged=true), log id: 63087612 >2014-09-09 15:50:20,951 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-6-thread-10) [6a92276b] FINISH, SetVdsStatusVDSCommand, log id: 63087612 >2014-09-09 15:50:21,148 INFO [org.ovirt.engine.core.bll.MaintenanceVdsCommand] (org.ovirt.thread.pool-6-thread-10) [6a92276b] Running command: MaintenanceVdsCommand internal: true. Entities affected : ID: 14369bd0-907d-40d7-be1e-bd35af6d1488 Type: VDS >2014-09-09 15:50:21,165 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-10) [6a92276b] Correlation ID: 6a92276b, Job ID: 28c8875a-05cd-4d63-998b-c03fb8c5a610, Call Stack: null, Custom Event ID: -1, Message: Host rack was switched to Maintenance mode by admin. >2014-09-09 15:50:23,753 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-38) Updated vds status from Preparing for Maintenance to Maintenance in database, vds = 14369bd0-907d-40d7-be1e-bd35af6d1488 : rack >2014-09-09 15:50:23,862 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-6-thread-12) Clearing cache of pool: 00000002-0002-0002-0002-0000000001ad for problematic entities of VDS: rack. >2014-09-09 15:50:23,863 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-6-thread-12) Removing vds [14369bd0-907d-40d7-be1e-bd35af6d1488] from the domain in maintenance cache >2014-09-09 15:50:23,863 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-6-thread-12) Removing host(s) [14369bd0-907d-40d7-be1e-bd35af6d1488] from hosts unseen domain report cache >2014-09-09 15:51:31,869 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.Reactor] (SSL Stomp Reactor) Unable to process messages: java.io.IOException: Connection reset by peer > at sun.nio.ch.FileDispatcherImpl.read0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39) [rt.jar:1.7.0_65] > at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223) [rt.jar:1.7.0_65] > at sun.nio.ch.IOUtil.read(IOUtil.java:197) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:379) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.SSLEngineNioHelper.read(SSLEngineNioHelper.java:51) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.SSLClient.read(SSLClient.java:81) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.readBuffer(ReactorClient.java:203) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.stomp.StompCommonClient.processIncoming(StompCommonClient.java:90) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.process(ReactorClient.java:144) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.SSLClient.process(SSLClient.java:115) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.processChannels(Reactor.java:86) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:62) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 15:57:22,257 INFO [org.ovirt.engine.core.bll.RemoveVdsCommand] (ajp-/127.0.0.1:8702-10) [337bc998] Lock Acquired to object EngineLock [exclusiveLocks= key: 14369bd0-907d-40d7-be1e-bd35af6d1488 value: VDS >, sharedLocks= ] >2014-09-09 15:57:22,304 INFO [org.ovirt.engine.core.bll.RemoveVdsCommand] (org.ovirt.thread.pool-6-thread-14) [337bc998] Running command: RemoveVdsCommand internal: false. Entities affected : ID: 14369bd0-907d-40d7-be1e-bd35af6d1488 Type: VDSAction group DELETE_HOST with role type ADMIN >2014-09-09 15:57:22,409 INFO [org.ovirt.engine.core.vdsbroker.RemoveVdsVDSCommand] (org.ovirt.thread.pool-6-thread-14) [337bc998] START, RemoveVdsVDSCommand( HostId = 14369bd0-907d-40d7-be1e-bd35af6d1488), log id: 6cfabc4 >2014-09-09 15:57:22,409 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (org.ovirt.thread.pool-6-thread-14) [337bc998] vdsManager::disposing >2014-09-09 15:57:22,419 INFO [org.ovirt.engine.core.vdsbroker.RemoveVdsVDSCommand] (org.ovirt.thread.pool-6-thread-14) [337bc998] FINISH, RemoveVdsVDSCommand, log id: 6cfabc4 >2014-09-09 15:57:22,443 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-14) [337bc998] Correlation ID: 337bc998, Call Stack: null, Custom Event ID: -1, Message: Host rack was removed by admin. >2014-09-09 15:57:22,445 INFO [org.ovirt.engine.core.bll.RemoveVdsCommand] (org.ovirt.thread.pool-6-thread-14) [337bc998] Lock freed to object EngineLock [exclusiveLocks= key: 14369bd0-907d-40d7-be1e-bd35af6d1488 value: VDS >, sharedLocks= ] >2014-09-09 16:24:37,443 INFO [org.ovirt.engine.core.bll.AddVdsCommand] (ajp-/127.0.0.1:8702-7) [1f633769] Running command: AddVdsCommand internal: false. Entities affected : ID: 00000001-0001-0001-0001-000000000399 Type: VdsGroupsAction group CREATE_HOST with role type ADMIN >2014-09-09 16:24:37,640 INFO [org.ovirt.engine.core.bll.AddVdsSpmIdCommand] (ajp-/127.0.0.1:8702-7) [207f962e] Running command: AddVdsSpmIdCommand internal: true. Entities affected : ID: 1fc56aac-07f1-49db-8972-b5b9567dee77 Type: VDS >2014-09-09 16:24:37,687 INFO [org.ovirt.engine.core.bll.AddVdsSpmIdCommand] (ajp-/127.0.0.1:8702-7) [207f962e] Lock freed to object EngineLock [exclusiveLocks= key: 00000002-0002-0002-0002-0000000001ad value: REGISTER_VDS >, sharedLocks= ] >2014-09-09 16:24:37,691 INFO [org.ovirt.engine.core.vdsbroker.RemoveVdsVDSCommand] (ajp-/127.0.0.1:8702-7) [207f962e] START, RemoveVdsVDSCommand(HostName = rack, HostId = 1fc56aac-07f1-49db-8972-b5b9567dee77), log id: 3f5d2881 >2014-09-09 16:24:37,691 INFO [org.ovirt.engine.core.vdsbroker.RemoveVdsVDSCommand] (ajp-/127.0.0.1:8702-7) [207f962e] FINISH, RemoveVdsVDSCommand, log id: 3f5d2881 >2014-09-09 16:24:37,694 INFO [org.ovirt.engine.core.vdsbroker.AddVdsVDSCommand] (ajp-/127.0.0.1:8702-7) [207f962e] START, AddVdsVDSCommand(HostName = rack, HostId = 1fc56aac-07f1-49db-8972-b5b9567dee77), log id: 3e80e999 >2014-09-09 16:24:37,695 INFO [org.ovirt.engine.core.vdsbroker.AddVdsVDSCommand] (ajp-/127.0.0.1:8702-7) [207f962e] AddVds - entered , starting logic to add VDS 1fc56aac-07f1-49db-8972-b5b9567dee77 >2014-09-09 16:24:37,697 INFO [org.ovirt.engine.core.vdsbroker.AddVdsVDSCommand] (ajp-/127.0.0.1:8702-7) [207f962e] AddVds - VDS 1fc56aac-07f1-49db-8972-b5b9567dee77 was added, will try to add it to the resource manager >2014-09-09 16:24:37,697 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (ajp-/127.0.0.1:8702-7) [207f962e] Entered VdsManager constructor >2014-09-09 16:24:37,719 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (ajp-/127.0.0.1:8702-7) [207f962e] Initialize vdsBroker (dell-r210ii-16.rhev.lab.eng.brq.redhat.com,54,321) >2014-09-09 16:24:37,720 INFO [org.ovirt.engine.core.vdsbroker.ResourceManager] (ajp-/127.0.0.1:8702-7) [207f962e] VDS 1fc56aac-07f1-49db-8972-b5b9567dee77 was added to the Resource Manager >2014-09-09 16:24:37,720 INFO [org.ovirt.engine.core.vdsbroker.AddVdsVDSCommand] (ajp-/127.0.0.1:8702-7) [207f962e] FINISH, AddVdsVDSCommand, log id: 3e80e999 >2014-09-09 16:24:37,743 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-7) [207f962e] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host rack. >2014-09-09 16:24:37,852 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-7) [207f962e] Correlation ID: 1f633769, Job ID: ccf74cb2-cf32-4077-b13a-ace5a1f14e3f, Call Stack: null, Custom Event ID: -1, Message: Host rack was added by admin. >2014-09-09 16:24:37,854 INFO [org.ovirt.engine.core.bll.InstallVdsInternalCommand] (org.ovirt.thread.pool-6-thread-15) [647ba2dc] Lock Acquired to object EngineLock [exclusiveLocks= key: 1fc56aac-07f1-49db-8972-b5b9567dee77 value: VDS >, sharedLocks= ] >2014-09-09 16:24:37,885 INFO [org.ovirt.engine.core.bll.InstallVdsInternalCommand] (org.ovirt.thread.pool-6-thread-15) [647ba2dc] Running command: InstallVdsInternalCommand internal: true. Entities affected : ID: 1fc56aac-07f1-49db-8972-b5b9567dee77 Type: VDS >2014-09-09 16:24:37,893 INFO [org.ovirt.engine.core.bll.InstallVdsInternalCommand] (org.ovirt.thread.pool-6-thread-15) [647ba2dc] Before Installation host 1fc56aac-07f1-49db-8972-b5b9567dee77, rack >2014-09-09 16:24:37,902 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-6-thread-15) [647ba2dc] START, SetVdsStatusVDSCommand(HostName = rack, HostId = 1fc56aac-07f1-49db-8972-b5b9567dee77, status=Installing, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 38d9f719 >2014-09-09 16:24:37,915 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-6-thread-15) [647ba2dc] FINISH, SetVdsStatusVDSCommand, log id: 38d9f719 >2014-09-09 16:24:37,953 INFO [org.ovirt.engine.core.bll.InstallerMessages] (org.ovirt.thread.pool-6-thread-15) [647ba2dc] Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Connected to host dell-r210ii-16.rhev.lab.eng.brq.redhat.com with SSH key fingerprint: 3f:fe:39:f2:21:7b:9c:a0:53:49:a6:35:b0:6c:4a:ff >2014-09-09 16:24:37,971 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-15) [647ba2dc] Correlation ID: 647ba2dc, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Connected to host dell-r210ii-16.rhev.lab.eng.brq.redhat.com with SSH key fingerprint: 3f:fe:39:f2:21:7b:9c:a0:53:49:a6:35:b0:6c:4a:ff. >2014-09-09 16:24:38,032 INFO [org.ovirt.engine.core.bll.VdsDeploy] (org.ovirt.thread.pool-6-thread-15) [647ba2dc] Installation of dell-r210ii-16.rhev.lab.eng.brq.redhat.com. Executing command via SSH umask 0077; MYTMP="$(TMPDIR="${OVIRT_TMPDIR}" mktemp -t ovirt-XXXXXXXXXX)"; trap "chmod -R u+rwX \"${MYTMP}\" > /dev/null 2>&1; rm -fr \"${MYTMP}\" > /dev/null 2>&1" 0; rm -fr "${MYTMP}" && mkdir "${MYTMP}" && tar --warning=no-timestamp -C "${MYTMP}" -x && "${MYTMP}"/setup DIALOG/dialect=str:machine DIALOG/customization=bool:True < /var/cache/ovirt-engine/ovirt-host-deploy.tar >2014-09-09 16:24:38,045 INFO [org.ovirt.engine.core.uutils.ssh.SSHDialog] (org.ovirt.thread.pool-6-thread-15) SSH execute root@dell-r210ii-16.rhev.lab.eng.brq.redhat.com 'umask 0077; MYTMP="$(TMPDIR="${OVIRT_TMPDIR}" mktemp -t ovirt-XXXXXXXXXX)"; trap "chmod -R u+rwX \"${MYTMP}\" > /dev/null 2>&1; rm -fr \"${MYTMP}\" > /dev/null 2>&1" 0; rm -fr "${MYTMP}" && mkdir "${MYTMP}" && tar --warning=no-timestamp -C "${MYTMP}" -x && "${MYTMP}"/setup DIALOG/dialect=str:machine DIALOG/customization=bool:True' >2014-09-09 16:24:40,724 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-44) Initializing Host: rack >2014-09-09 16:24:40,913 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Stage: Initializing >2014-09-09 16:24:40,935 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 647ba2dc, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Stage: Initializing. >2014-09-09 16:24:40,960 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Stage: Environment setup >2014-09-09 16:24:40,989 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 647ba2dc, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Stage: Environment setup. >2014-09-09 16:24:40,989 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Stage: Environment packages setup >2014-09-09 16:24:41,027 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 647ba2dc, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Stage: Environment packages setup. >2014-09-09 16:24:41,027 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Stage: Programs detection >2014-09-09 16:24:41,044 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 647ba2dc, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Stage: Programs detection. >2014-09-09 16:24:41,044 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Stage: Environment customization >2014-09-09 16:24:41,064 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 647ba2dc, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Stage: Environment customization. >2014-09-09 16:24:41,064 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Kdump unsupported >2014-09-09 16:24:41,086 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 647ba2dc, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Kdump unsupported. >2014-09-09 16:24:41,102 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Logs at host located at: '/tmp/ovirt-host-deploy-20140909162438-1rvbah.log' >2014-09-09 16:24:41,116 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 647ba2dc, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Logs at host located at: '/tmp/ovirt-host-deploy-20140909162438-1rvbah.log'. >2014-09-09 16:24:41,234 INFO [org.ovirt.engine.core.bll.VdsDeploy] (VdsDeploy) Host dell-r210ii-16.rhev.lab.eng.brq.redhat.com reports unique id 4C4C4544-0034-5310-8052-B4C04F4A354A >2014-09-09 16:24:41,240 INFO [org.ovirt.engine.core.bll.VdsDeploy] (VdsDeploy) Assigning unique id 4C4C4544-0034-5310-8052-B4C04F4A354A to Host dell-r210ii-16.rhev.lab.eng.brq.redhat.com >2014-09-09 16:24:41,357 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Disabling Kdump integration >2014-09-09 16:24:41,383 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 647ba2dc, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Disabling Kdump integration. >2014-09-09 16:24:41,416 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Stage: Setup validation >2014-09-09 16:24:41,433 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 647ba2dc, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Stage: Setup validation. >2014-09-09 16:24:41,453 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Hardware supports virtualization >2014-09-09 16:24:41,468 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 647ba2dc, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Hardware supports virtualization. >2014-09-09 16:24:41,469 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Stage: Transaction setup >2014-09-09 16:24:41,480 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 647ba2dc, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Stage: Transaction setup. >2014-09-09 16:24:41,480 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Stage: Misc configuration >2014-09-09 16:24:41,491 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 647ba2dc, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Stage: Misc configuration. >2014-09-09 16:24:41,492 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Stage: Package installation >2014-09-09 16:24:41,506 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 647ba2dc, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Stage: Package installation. >2014-09-09 16:24:48,013 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Stage: Misc configuration >2014-09-09 16:24:48,026 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 647ba2dc, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Stage: Misc configuration. >2014-09-09 16:24:49,120 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Enrolling certificate >2014-09-09 16:24:49,133 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 647ba2dc, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Enrolling certificate. >2014-09-09 16:24:50,416 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Stage: Transaction commit >2014-09-09 16:24:50,429 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 647ba2dc, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Stage: Transaction commit. >2014-09-09 16:24:50,429 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Stage: Closing up >2014-09-09 16:24:50,441 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 647ba2dc, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Stage: Closing up. >2014-09-09 16:24:51,246 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Stopping libvirtd >2014-09-09 16:24:51,258 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 647ba2dc, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Stopping libvirtd. >2014-09-09 16:24:51,588 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Starting vdsm >2014-09-09 16:24:51,600 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 647ba2dc, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Starting vdsm. >2014-09-09 16:24:53,355 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Stage: Pre-termination >2014-09-09 16:24:53,372 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 647ba2dc, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Stage: Pre-termination. >2014-09-09 16:24:53,389 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Retrieving installation logs to: '/var/log/ovirt-engine/host-deploy/ovirt-20140909162453-dell-r210ii-16.rhev.lab.eng.brq.redhat.com-647ba2dc.log' >2014-09-09 16:24:53,407 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 647ba2dc, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Retrieving installation logs to: '/var/log/ovirt-engine/host-deploy/ovirt-20140909162453-dell-r210ii-16.rhev.lab.eng.brq.redhat.com-647ba2dc.log'. >2014-09-09 16:24:53,708 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Stage: Termination >2014-09-09 16:24:53,720 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 647ba2dc, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Stage: Termination. >2014-09-09 16:24:53,855 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to dell-r210ii-16.rhev.lab.eng.brq.redhat.com/10.34.62.208 >2014-09-09 16:24:54,010 INFO [org.ovirt.engine.core.bll.network.NetworkConfigurator] (org.ovirt.thread.pool-6-thread-15) [647ba2dc] Engine managed to communicate with VDSM agent on host rack >2014-09-09 16:24:54,687 WARN [org.ovirt.engine.core.bll.network.NetworkConfigurator] (org.ovirt.thread.pool-6-thread-15) [647ba2dc] The lastClientInterface ovirtmgmt of host rack is not a valid interface for the mangement network. If the interface is a bridge, it should be torn-down manually. >2014-09-09 16:24:54,693 ERROR [org.ovirt.engine.core.bll.InstallVdsInternalCommand] (org.ovirt.thread.pool-6-thread-15) [647ba2dc] Host installation failed for host 1fc56aac-07f1-49db-8972-b5b9567dee77, rack.: org.ovirt.engine.core.bll.VdsCommand$VdsInstallException: Failed to configure management network on the host > at org.ovirt.engine.core.bll.InstallVdsInternalCommand.configureManagementNetwork(InstallVdsInternalCommand.java:206) [bll.jar:] > at org.ovirt.engine.core.bll.InstallVdsInternalCommand.installHost(InstallVdsInternalCommand.java:164) [bll.jar:] > at org.ovirt.engine.core.bll.InstallVdsInternalCommand.executeCommand(InstallVdsInternalCommand.java:78) [bll.jar:] > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1167) [bll.jar:] > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1298) [bll.jar:] > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1920) [bll.jar:] > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) [utils.jar:] > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) [utils.jar:] > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1319) [bll.jar:] > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:344) [bll.jar:] > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:430) [bll.jar:] > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:411) [bll.jar:] > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:616) [bll.jar:] > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_65] > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptor.java:52) [jboss-as-ee.jar:7.4.0.Final-redhat-19] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:63) [jboss-as-ee.jar:7.4.0.Final-redhat-19] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:374) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.as.weld.ejb.Jsr299BindingsInterceptor.delegateInterception(Jsr299BindingsInterceptor.java:74) [jboss-as-weld.jar:7.4.0.Final-redhat-19] > at org.jboss.as.weld.ejb.Jsr299BindingsInterceptor.doMethodInterception(Jsr299BindingsInterceptor.java:84) [jboss-as-weld.jar:7.4.0.Final-redhat-19] > at org.jboss.as.weld.ejb.Jsr299BindingsInterceptor.processInvocation(Jsr299BindingsInterceptor.java:97) [jboss-as-weld.jar:7.4.0.Final-redhat-19] > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:63) [jboss-as-ee.jar:7.4.0.Final-redhat-19] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:63) [jboss-as-ee.jar:7.4.0.Final-redhat-19] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) [jboss-as-ejb3.jar:7.4.0.Final-redhat-19] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.as.weld.ejb.EjbRequestScopeActivationInterceptor.processInvocation(EjbRequestScopeActivationInterceptor.java:93) [jboss-as-weld.jar:7.4.0.Final-redhat-19] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) [jboss-as-ee.jar:7.4.0.Final-redhat-19] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) [jboss-as-ejb3.jar:7.4.0.Final-redhat-19] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) [jboss-as-ejb3.jar:7.4.0.Final-redhat-19] > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) [jboss-as-ejb3.jar:7.4.0.Final-redhat-19] > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) [jboss-as-ejb3.jar:7.4.0.Final-redhat-19] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) [jboss-as-ejb3.jar:7.4.0.Final-redhat-19] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) [jboss-as-ejb3.jar:7.4.0.Final-redhat-19] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) [jboss-as-ejb3.jar:7.4.0.Final-redhat-19] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) [jboss-as-ee.jar:7.4.0.Final-redhat-19] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) [jboss-as-ee.jar:7.4.0.Final-redhat-19] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:185) [jboss-as-ee.jar:7.4.0.Final-redhat-19] > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) [jboss-as-ee.jar:7.4.0.Final-redhat-19] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:73) [jboss-as-ee.jar:7.4.0.Final-redhat-19] > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) [bll.jar:] > at org.ovirt.engine.core.bll.CommandBase.runInternalAction(CommandBase.java:2249) [bll.jar:] > at org.ovirt.engine.core.bll.AddVdsCommand$3.run(AddVdsCommand.java:195) [bll.jar:] > at org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil$InternalWrapperRunnable.run(ThreadPoolUtil.java:90) [utils.jar:] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_65] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_65] > at java.lang.Thread.run(Thread.java:745) [rt.jar:1.7.0_65] >Caused by: org.ovirt.engine.core.bll.network.NetworkConfigurator$NetworkConfiguratorException: lastClientIface ovirtmgmt is not a valid interface for management network > at org.ovirt.engine.core.bll.network.NetworkConfigurator.findNicToSetupManagementNetwork(NetworkConfigurator.java:170) [bll.jar:] > at org.ovirt.engine.core.bll.network.NetworkConfigurator.createManagementNetworkIfRequired(NetworkConfigurator.java:74) [bll.jar:] > at org.ovirt.engine.core.bll.InstallVdsInternalCommand.configureManagementNetwork(InstallVdsInternalCommand.java:198) [bll.jar:] > ... 70 more > >2014-09-09 16:24:54,702 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-6-thread-15) [647ba2dc] START, SetVdsStatusVDSCommand(HostName = rack, HostId = 1fc56aac-07f1-49db-8972-b5b9567dee77, status=NonOperational, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 242d4a81 >2014-09-09 16:24:54,727 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-6-thread-15) [647ba2dc] FINISH, SetVdsStatusVDSCommand, log id: 242d4a81 >2014-09-09 16:24:54,770 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-15) [647ba2dc] Correlation ID: 647ba2dc, Job ID: ccf74cb2-cf32-4077-b13a-ace5a1f14e3f, Call Stack: null, Custom Event ID: -1, Message: Host rack installation failed. Failed to configure management network on the host. >2014-09-09 16:24:54,782 INFO [org.ovirt.engine.core.bll.InstallVdsInternalCommand] (org.ovirt.thread.pool-6-thread-15) [647ba2dc] Lock freed to object EngineLock [exclusiveLocks= key: 1fc56aac-07f1-49db-8972-b5b9567dee77 value: VDS >, sharedLocks= ] >2014-09-09 16:25:00,062 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-90) Autorecovering 1 hosts >2014-09-09 16:25:00,062 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-90) Autorecovering hosts id: 1fc56aac-07f1-49db-8972-b5b9567dee77, name : rack >2014-09-09 16:25:00,069 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-90) [18a5c631] Lock Acquired to object EngineLock [exclusiveLocks= key: 1fc56aac-07f1-49db-8972-b5b9567dee77 value: VDS >, sharedLocks= ] >2014-09-09 16:25:00,069 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-90) [18a5c631] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 1fc56aac-07f1-49db-8972-b5b9567dee77 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2014-09-09 16:25:00,069 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-90) [18a5c631] Before acquiring lock in order to prevent monitoring for host rack from data-center Default >2014-09-09 16:25:00,069 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-90) [18a5c631] Lock acquired, from now a monitoring of host will be skipped for host rack from data-center Default >2014-09-09 16:25:00,074 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-90) [18a5c631] START, SetVdsStatusVDSCommand(HostName = rack, HostId = 1fc56aac-07f1-49db-8972-b5b9567dee77, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 45ec3e9d >2014-09-09 16:25:00,101 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-90) [18a5c631] FINISH, SetVdsStatusVDSCommand, log id: 45ec3e9d >2014-09-09 16:25:00,134 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-90) [18a5c631] Activate finished. Lock released. Monitoring can run now for host rack from data-center Default >2014-09-09 16:25:00,146 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-90) [18a5c631] Correlation ID: 18a5c631, Call Stack: null, Custom Event ID: -1, Message: Host rack was autorecovered. >2014-09-09 16:25:00,146 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-90) [18a5c631] Lock freed to object EngineLock [exclusiveLocks= key: 1fc56aac-07f1-49db-8972-b5b9567dee77 value: VDS >, sharedLocks= ] >2014-09-09 16:25:01,876 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-95) START, GetHardwareInfoVDSCommand(HostName = rack, HostId = 1fc56aac-07f1-49db-8972-b5b9567dee77, vds=Host[rack,1fc56aac-07f1-49db-8972-b5b9567dee77]), log id: 6b0a9c81 >2014-09-09 16:25:01,969 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-95) FINISH, GetHardwareInfoVDSCommand, log id: 6b0a9c81 >2014-09-09 16:25:02,082 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-95) [3feefd64] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 1fc56aac-07f1-49db-8972-b5b9567dee77 Type: VDS >2014-09-09 16:25:02,086 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-95) [3feefd64] START, SetVdsStatusVDSCommand(HostName = rack, HostId = 1fc56aac-07f1-49db-8972-b5b9567dee77, status=NonOperational, nonOperationalReason=NETWORK_UNREACHABLE, stopSpmFailureLogged=false), log id: 3ff7b6f9 >2014-09-09 16:25:02,111 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-95) [3feefd64] FINISH, SetVdsStatusVDSCommand, log id: 3ff7b6f9 >2014-09-09 16:25:02,116 ERROR [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-95) [3feefd64] Host rack is set to Non-Operational, it is missing the following networks: rhevm >2014-09-09 16:25:02,139 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-95) [3feefd64] Correlation ID: 3feefd64, Job ID: 4a0b2277-d850-4deb-b2df-b1c10ef2145e, Call Stack: null, Custom Event ID: -1, Message: Host rack does not comply with the cluster Default networks, the following networks are missing on host: 'rhevm' >2014-09-09 16:25:02,163 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-95) [5f392203] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 1fc56aac-07f1-49db-8972-b5b9567dee77 Type: VDS >2014-09-09 16:25:02,203 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-95) [5f392203] START, GetHardwareInfoVDSCommand(HostName = rack, HostId = 1fc56aac-07f1-49db-8972-b5b9567dee77, vds=Host[rack,1fc56aac-07f1-49db-8972-b5b9567dee77]), log id: 403a6004 >2014-09-09 16:25:02,208 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-95) [5f392203] FINISH, GetHardwareInfoVDSCommand, log id: 403a6004 >2014-09-09 16:25:02,288 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-95) [1d0acec4] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 1fc56aac-07f1-49db-8972-b5b9567dee77 Type: VDS >2014-09-09 16:25:02,291 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-95) [1d0acec4] START, SetVdsStatusVDSCommand(HostName = rack, HostId = 1fc56aac-07f1-49db-8972-b5b9567dee77, status=NonOperational, nonOperationalReason=NETWORK_UNREACHABLE, stopSpmFailureLogged=false), log id: 6a7e3293 >2014-09-09 16:25:02,302 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-95) [1d0acec4] FINISH, SetVdsStatusVDSCommand, log id: 6a7e3293 >2014-09-09 16:25:02,336 ERROR [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-95) [1d0acec4] Host rack is set to Non-Operational, it is missing the following networks: rhevm >2014-09-09 16:25:02,354 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-95) [6eafe1f4] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 1fc56aac-07f1-49db-8972-b5b9567dee77 Type: VDS >2014-09-09 16:25:02,369 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-95) [6eafe1f4] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host rack was set to NonOperational. >2014-09-09 16:25:02,409 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-95) [13c332f3] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 1fc56aac-07f1-49db-8972-b5b9567dee77 Type: VDS >2014-09-09 16:25:02,414 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-95) [24a2c919] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 1fc56aac-07f1-49db-8972-b5b9567dee77 Type: VDS >2014-09-09 16:25:02,416 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-95) [24a2c919] Host 1fc56aac-07f1-49db-8972-b5b9567dee77 : rack is already in NonOperational status for reason NETWORK_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-09-09 16:30:00,020 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-69) Autorecovering 1 hosts >2014-09-09 16:30:00,020 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-69) Autorecovering hosts id: 1fc56aac-07f1-49db-8972-b5b9567dee77, name : rack >2014-09-09 16:30:00,025 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-69) [2f5064a8] Lock Acquired to object EngineLock [exclusiveLocks= key: 1fc56aac-07f1-49db-8972-b5b9567dee77 value: VDS >, sharedLocks= ] >2014-09-09 16:30:00,025 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-69) [2f5064a8] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 1fc56aac-07f1-49db-8972-b5b9567dee77 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2014-09-09 16:30:00,026 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-69) [2f5064a8] Before acquiring lock in order to prevent monitoring for host rack from data-center Default >2014-09-09 16:30:00,026 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-69) [2f5064a8] Lock acquired, from now a monitoring of host will be skipped for host rack from data-center Default >2014-09-09 16:30:00,035 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-69) [2f5064a8] START, SetVdsStatusVDSCommand(HostName = rack, HostId = 1fc56aac-07f1-49db-8972-b5b9567dee77, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 24310dd8 >2014-09-09 16:30:00,070 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-69) [2f5064a8] FINISH, SetVdsStatusVDSCommand, log id: 24310dd8 >2014-09-09 16:30:00,137 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-69) [2f5064a8] Activate finished. Lock released. Monitoring can run now for host rack from data-center Default >2014-09-09 16:30:00,140 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-69) [2f5064a8] Lock freed to object EngineLock [exclusiveLocks= key: 1fc56aac-07f1-49db-8972-b5b9567dee77 value: VDS >, sharedLocks= ] >2014-09-09 16:30:03,029 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-77) START, GetHardwareInfoVDSCommand(HostName = rack, HostId = 1fc56aac-07f1-49db-8972-b5b9567dee77, vds=Host[rack,1fc56aac-07f1-49db-8972-b5b9567dee77]), log id: 61da0aea >2014-09-09 16:30:03,036 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-77) FINISH, GetHardwareInfoVDSCommand, log id: 61da0aea >2014-09-09 16:30:03,132 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-77) [5460fbba] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 1fc56aac-07f1-49db-8972-b5b9567dee77 Type: VDS >2014-09-09 16:30:03,136 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-77) [5460fbba] START, SetVdsStatusVDSCommand(HostName = rack, HostId = 1fc56aac-07f1-49db-8972-b5b9567dee77, status=NonOperational, nonOperationalReason=NETWORK_UNREACHABLE, stopSpmFailureLogged=false), log id: 2cbced61 >2014-09-09 16:30:03,182 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-77) [5460fbba] FINISH, SetVdsStatusVDSCommand, log id: 2cbced61 >2014-09-09 16:30:03,190 ERROR [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-77) [5460fbba] Host rack is set to Non-Operational, it is missing the following networks: rhevm >2014-09-09 16:30:03,215 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-77) [5460fbba] Correlation ID: 5460fbba, Job ID: 47271c9e-6cdd-4f5c-8ea4-631b37faff33, Call Stack: null, Custom Event ID: -1, Message: Host rack does not comply with the cluster Default networks, the following networks are missing on host: 'rhevm' >2014-09-09 16:30:03,237 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-77) [4275b3b] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 1fc56aac-07f1-49db-8972-b5b9567dee77 Type: VDS >2014-09-09 16:30:03,257 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-77) [4275b3b] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host rack was set to NonOperational. >2014-09-09 16:30:03,302 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-77) [1d4fb057] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 1fc56aac-07f1-49db-8972-b5b9567dee77 Type: VDS >2014-09-09 16:30:03,305 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-77) [1d4fb057] Host 1fc56aac-07f1-49db-8972-b5b9567dee77 : rack is already in NonOperational status for reason NETWORK_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-09-09 16:35:00,030 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-48) Autorecovering 1 hosts >2014-09-09 16:35:00,030 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-48) Autorecovering hosts id: 1fc56aac-07f1-49db-8972-b5b9567dee77, name : rack >2014-09-09 16:35:00,035 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-48) [1a5d7bf6] Lock Acquired to object EngineLock [exclusiveLocks= key: 1fc56aac-07f1-49db-8972-b5b9567dee77 value: VDS >, sharedLocks= ] >2014-09-09 16:35:00,035 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-48) [1a5d7bf6] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 1fc56aac-07f1-49db-8972-b5b9567dee77 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2014-09-09 16:35:00,035 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-48) [1a5d7bf6] Before acquiring lock in order to prevent monitoring for host rack from data-center Default >2014-09-09 16:35:00,035 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-48) [1a5d7bf6] Lock acquired, from now a monitoring of host will be skipped for host rack from data-center Default >2014-09-09 16:35:00,045 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-48) [1a5d7bf6] START, SetVdsStatusVDSCommand(HostName = rack, HostId = 1fc56aac-07f1-49db-8972-b5b9567dee77, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 19846f38 >2014-09-09 16:35:00,073 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-48) [1a5d7bf6] FINISH, SetVdsStatusVDSCommand, log id: 19846f38 >2014-09-09 16:35:00,087 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-48) [1a5d7bf6] Activate finished. Lock released. Monitoring can run now for host rack from data-center Default >2014-09-09 16:35:00,090 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-48) [1a5d7bf6] Lock freed to object EngineLock [exclusiveLocks= key: 1fc56aac-07f1-49db-8972-b5b9567dee77 value: VDS >, sharedLocks= ] >2014-09-09 16:35:00,868 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-49) START, GetHardwareInfoVDSCommand(HostName = rack, HostId = 1fc56aac-07f1-49db-8972-b5b9567dee77, vds=Host[rack,1fc56aac-07f1-49db-8972-b5b9567dee77]), log id: 6e050268 >2014-09-09 16:35:00,873 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-49) FINISH, GetHardwareInfoVDSCommand, log id: 6e050268 >2014-09-09 16:35:00,980 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-49) [69dd153d] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 1fc56aac-07f1-49db-8972-b5b9567dee77 Type: VDS >2014-09-09 16:35:00,984 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-49) [69dd153d] START, SetVdsStatusVDSCommand(HostName = rack, HostId = 1fc56aac-07f1-49db-8972-b5b9567dee77, status=NonOperational, nonOperationalReason=NETWORK_UNREACHABLE, stopSpmFailureLogged=false), log id: 95fabe0 >2014-09-09 16:35:01,013 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-49) [69dd153d] FINISH, SetVdsStatusVDSCommand, log id: 95fabe0 >2014-09-09 16:35:01,020 ERROR [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-49) [69dd153d] Host rack is set to Non-Operational, it is missing the following networks: rhevm >2014-09-09 16:35:01,042 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-49) [69dd153d] Correlation ID: 69dd153d, Job ID: fd42f06a-cf2e-49b2-8b1d-c141bd88a4be, Call Stack: null, Custom Event ID: -1, Message: Host rack does not comply with the cluster Default networks, the following networks are missing on host: 'rhevm' >2014-09-09 16:35:01,065 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-49) [7fec9ac8] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 1fc56aac-07f1-49db-8972-b5b9567dee77 Type: VDS >2014-09-09 16:35:01,086 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-49) [7fec9ac8] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host rack was set to NonOperational. >2014-09-09 16:35:01,133 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-49) [7e37bc27] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 1fc56aac-07f1-49db-8972-b5b9567dee77 Type: VDS >2014-09-09 16:35:01,137 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-49) [7e37bc27] Host 1fc56aac-07f1-49db-8972-b5b9567dee77 : rack is already in NonOperational status for reason NETWORK_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-09-09 16:40:00,039 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-25) Autorecovering 1 hosts >2014-09-09 16:40:00,039 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-25) Autorecovering hosts id: 1fc56aac-07f1-49db-8972-b5b9567dee77, name : rack >2014-09-09 16:40:00,052 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-25) [27472998] Lock Acquired to object EngineLock [exclusiveLocks= key: 1fc56aac-07f1-49db-8972-b5b9567dee77 value: VDS >, sharedLocks= ] >2014-09-09 16:40:00,053 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-25) [27472998] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 1fc56aac-07f1-49db-8972-b5b9567dee77 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2014-09-09 16:40:00,053 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-25) [27472998] Before acquiring lock in order to prevent monitoring for host rack from data-center Default >2014-09-09 16:40:00,053 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-25) [27472998] Lock acquired, from now a monitoring of host will be skipped for host rack from data-center Default >2014-09-09 16:40:00,064 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-25) [27472998] START, SetVdsStatusVDSCommand(HostName = rack, HostId = 1fc56aac-07f1-49db-8972-b5b9567dee77, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 28b2ecf7 >2014-09-09 16:40:00,112 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-25) [27472998] FINISH, SetVdsStatusVDSCommand, log id: 28b2ecf7 >2014-09-09 16:40:00,154 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-25) [27472998] Activate finished. Lock released. Monitoring can run now for host rack from data-center Default >2014-09-09 16:40:00,156 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-25) [27472998] Lock freed to object EngineLock [exclusiveLocks= key: 1fc56aac-07f1-49db-8972-b5b9567dee77 value: VDS >, sharedLocks= ] >2014-09-09 16:40:01,653 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-33) START, GetHardwareInfoVDSCommand(HostName = rack, HostId = 1fc56aac-07f1-49db-8972-b5b9567dee77, vds=Host[rack,1fc56aac-07f1-49db-8972-b5b9567dee77]), log id: 3d24571c >2014-09-09 16:40:01,659 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-33) FINISH, GetHardwareInfoVDSCommand, log id: 3d24571c >2014-09-09 16:40:01,786 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-33) [51010aba] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 1fc56aac-07f1-49db-8972-b5b9567dee77 Type: VDS >2014-09-09 16:40:01,791 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-33) [51010aba] START, SetVdsStatusVDSCommand(HostName = rack, HostId = 1fc56aac-07f1-49db-8972-b5b9567dee77, status=NonOperational, nonOperationalReason=NETWORK_UNREACHABLE, stopSpmFailureLogged=false), log id: 48e7fb7b >2014-09-09 16:40:01,828 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-33) [51010aba] FINISH, SetVdsStatusVDSCommand, log id: 48e7fb7b >2014-09-09 16:40:01,836 ERROR [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-33) [51010aba] Host rack is set to Non-Operational, it is missing the following networks: rhevm >2014-09-09 16:40:01,856 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-33) [51010aba] Correlation ID: 51010aba, Job ID: 40057208-b2e7-4841-ae54-92c5bf9ecfd0, Call Stack: null, Custom Event ID: -1, Message: Host rack does not comply with the cluster Default networks, the following networks are missing on host: 'rhevm' >2014-09-09 16:40:01,883 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-33) [61109934] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 1fc56aac-07f1-49db-8972-b5b9567dee77 Type: VDS >2014-09-09 16:40:01,905 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-33) [61109934] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host rack was set to NonOperational. >2014-09-09 16:40:01,967 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-33) [8ff0b8] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 1fc56aac-07f1-49db-8972-b5b9567dee77 Type: VDS >2014-09-09 16:40:01,971 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-33) [8ff0b8] Host 1fc56aac-07f1-49db-8972-b5b9567dee77 : rack is already in NonOperational status for reason NETWORK_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-09-09 16:45:00,022 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-8) Autorecovering 1 hosts >2014-09-09 16:45:00,023 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-8) Autorecovering hosts id: 1fc56aac-07f1-49db-8972-b5b9567dee77, name : rack >2014-09-09 16:45:00,030 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-8) [28908be8] Lock Acquired to object EngineLock [exclusiveLocks= key: 1fc56aac-07f1-49db-8972-b5b9567dee77 value: VDS >, sharedLocks= ] >2014-09-09 16:45:00,030 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-8) [28908be8] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 1fc56aac-07f1-49db-8972-b5b9567dee77 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2014-09-09 16:45:00,030 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-8) [28908be8] Before acquiring lock in order to prevent monitoring for host rack from data-center Default >2014-09-09 16:45:00,031 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-8) [28908be8] Lock acquired, from now a monitoring of host will be skipped for host rack from data-center Default >2014-09-09 16:45:00,044 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-8) [28908be8] START, SetVdsStatusVDSCommand(HostName = rack, HostId = 1fc56aac-07f1-49db-8972-b5b9567dee77, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 246a5798 >2014-09-09 16:45:00,068 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-8) [28908be8] FINISH, SetVdsStatusVDSCommand, log id: 246a5798 >2014-09-09 16:45:00,081 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-8) [28908be8] Activate finished. Lock released. Monitoring can run now for host rack from data-center Default >2014-09-09 16:45:00,084 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-8) [28908be8] Lock freed to object EngineLock [exclusiveLocks= key: 1fc56aac-07f1-49db-8972-b5b9567dee77 value: VDS >, sharedLocks= ] >2014-09-09 16:45:02,501 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-10) START, GetHardwareInfoVDSCommand(HostName = rack, HostId = 1fc56aac-07f1-49db-8972-b5b9567dee77, vds=Host[rack,1fc56aac-07f1-49db-8972-b5b9567dee77]), log id: 78b8476a >2014-09-09 16:45:02,511 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-10) FINISH, GetHardwareInfoVDSCommand, log id: 78b8476a >2014-09-09 16:45:02,659 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-10) [51f90576] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 1fc56aac-07f1-49db-8972-b5b9567dee77 Type: VDS >2014-09-09 16:45:02,662 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-10) [51f90576] START, SetVdsStatusVDSCommand(HostName = rack, HostId = 1fc56aac-07f1-49db-8972-b5b9567dee77, status=NonOperational, nonOperationalReason=NETWORK_UNREACHABLE, stopSpmFailureLogged=false), log id: 7a06f81 >2014-09-09 16:45:02,691 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-10) [51f90576] FINISH, SetVdsStatusVDSCommand, log id: 7a06f81 >2014-09-09 16:45:02,749 ERROR [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-10) [51f90576] Host rack is set to Non-Operational, it is missing the following networks: rhevm >2014-09-09 16:45:02,777 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-10) [51f90576] Correlation ID: 51f90576, Job ID: 4ec8fe3f-42fc-44e8-9afe-4205fce8d3d3, Call Stack: null, Custom Event ID: -1, Message: Host rack does not comply with the cluster Default networks, the following networks are missing on host: 'rhevm' >2014-09-09 16:45:02,801 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-10) [7218fadb] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 1fc56aac-07f1-49db-8972-b5b9567dee77 Type: VDS >2014-09-09 16:45:02,819 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-10) [7218fadb] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host rack was set to NonOperational. >2014-09-09 16:45:02,873 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-10) [5a972ba0] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 1fc56aac-07f1-49db-8972-b5b9567dee77 Type: VDS >2014-09-09 16:45:02,876 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-10) [5a972ba0] Host 1fc56aac-07f1-49db-8972-b5b9567dee77 : rack is already in NonOperational status for reason NETWORK_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-09-09 16:47:52,248 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-94) Principal admin::internal synchronized >2014-09-09 16:47:52,587 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-95) [24a2c919] Attempting to update VMs/Templates Ovf. >2014-09-09 16:50:00,041 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-83) Autorecovering 1 hosts >2014-09-09 16:50:00,041 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-83) Autorecovering hosts id: 1fc56aac-07f1-49db-8972-b5b9567dee77, name : rack >2014-09-09 16:50:00,046 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-83) [b670564] Lock Acquired to object EngineLock [exclusiveLocks= key: 1fc56aac-07f1-49db-8972-b5b9567dee77 value: VDS >, sharedLocks= ] >2014-09-09 16:50:00,046 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-83) [b670564] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 1fc56aac-07f1-49db-8972-b5b9567dee77 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2014-09-09 16:50:00,046 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-83) [b670564] Before acquiring lock in order to prevent monitoring for host rack from data-center Default >2014-09-09 16:50:00,046 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-83) [b670564] Lock acquired, from now a monitoring of host will be skipped for host rack from data-center Default >2014-09-09 16:50:00,062 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-83) [b670564] START, SetVdsStatusVDSCommand(HostName = rack, HostId = 1fc56aac-07f1-49db-8972-b5b9567dee77, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 92438f6 >2014-09-09 16:50:00,097 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-83) [b670564] FINISH, SetVdsStatusVDSCommand, log id: 92438f6 >2014-09-09 16:50:00,140 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-83) [b670564] Activate finished. Lock released. Monitoring can run now for host rack from data-center Default >2014-09-09 16:50:00,144 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-83) [b670564] Lock freed to object EngineLock [exclusiveLocks= key: 1fc56aac-07f1-49db-8972-b5b9567dee77 value: VDS >, sharedLocks= ] >2014-09-09 16:50:00,399 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-89) START, GetHardwareInfoVDSCommand(HostName = rack, HostId = 1fc56aac-07f1-49db-8972-b5b9567dee77, vds=Host[rack,1fc56aac-07f1-49db-8972-b5b9567dee77]), log id: 2ebfecda >2014-09-09 16:50:00,404 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-89) FINISH, GetHardwareInfoVDSCommand, log id: 2ebfecda >2014-09-09 16:50:00,500 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-89) [7c825d3] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 1fc56aac-07f1-49db-8972-b5b9567dee77 Type: VDS >2014-09-09 16:50:00,504 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-89) [7c825d3] START, SetVdsStatusVDSCommand(HostName = rack, HostId = 1fc56aac-07f1-49db-8972-b5b9567dee77, status=NonOperational, nonOperationalReason=NETWORK_UNREACHABLE, stopSpmFailureLogged=false), log id: 34604fd6 >2014-09-09 16:50:00,535 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-89) [7c825d3] FINISH, SetVdsStatusVDSCommand, log id: 34604fd6 >2014-09-09 16:50:00,542 ERROR [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-89) [7c825d3] Host rack is set to Non-Operational, it is missing the following networks: rhevm >2014-09-09 16:50:00,563 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-89) [7c825d3] Correlation ID: 7c825d3, Job ID: 10891a4f-7345-4ea9-832c-c47f89ff1126, Call Stack: null, Custom Event ID: -1, Message: Host rack does not comply with the cluster Default networks, the following networks are missing on host: 'rhevm' >2014-09-09 16:50:00,584 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-89) [38ee5f8a] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 1fc56aac-07f1-49db-8972-b5b9567dee77 Type: VDS >2014-09-09 16:50:00,600 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-89) [38ee5f8a] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host rack was set to NonOperational. >2014-09-09 16:50:00,648 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-89) [55b965af] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 1fc56aac-07f1-49db-8972-b5b9567dee77 Type: VDS >2014-09-09 16:50:00,650 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-89) [55b965af] Host 1fc56aac-07f1-49db-8972-b5b9567dee77 : rack is already in NonOperational status for reason NETWORK_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-09-09 16:55:00,021 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-66) Autorecovering 1 hosts >2014-09-09 16:55:00,021 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-66) Autorecovering hosts id: 1fc56aac-07f1-49db-8972-b5b9567dee77, name : rack >2014-09-09 16:55:00,025 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-66) [266d3fbf] Lock Acquired to object EngineLock [exclusiveLocks= key: 1fc56aac-07f1-49db-8972-b5b9567dee77 value: VDS >, sharedLocks= ] >2014-09-09 16:55:00,025 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-66) [266d3fbf] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 1fc56aac-07f1-49db-8972-b5b9567dee77 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2014-09-09 16:55:00,026 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-66) [266d3fbf] Before acquiring lock in order to prevent monitoring for host rack from data-center Default >2014-09-09 16:55:00,026 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-66) [266d3fbf] Lock acquired, from now a monitoring of host will be skipped for host rack from data-center Default >2014-09-09 16:55:00,035 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-66) [266d3fbf] START, SetVdsStatusVDSCommand(HostName = rack, HostId = 1fc56aac-07f1-49db-8972-b5b9567dee77, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 3e4c81e7 >2014-09-09 16:55:00,072 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-66) [266d3fbf] FINISH, SetVdsStatusVDSCommand, log id: 3e4c81e7 >2014-09-09 16:55:00,088 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-66) [266d3fbf] Activate finished. Lock released. Monitoring can run now for host rack from data-center Default >2014-09-09 16:55:00,091 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-66) [266d3fbf] Lock freed to object EngineLock [exclusiveLocks= key: 1fc56aac-07f1-49db-8972-b5b9567dee77 value: VDS >, sharedLocks= ] >2014-09-09 16:55:01,168 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-72) START, GetHardwareInfoVDSCommand(HostName = rack, HostId = 1fc56aac-07f1-49db-8972-b5b9567dee77, vds=Host[rack,1fc56aac-07f1-49db-8972-b5b9567dee77]), log id: 37b6906 >2014-09-09 16:55:01,173 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-72) FINISH, GetHardwareInfoVDSCommand, log id: 37b6906 >2014-09-09 16:55:01,293 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-72) [64385954] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 1fc56aac-07f1-49db-8972-b5b9567dee77 Type: VDS >2014-09-09 16:55:01,299 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-72) [64385954] START, SetVdsStatusVDSCommand(HostName = rack, HostId = 1fc56aac-07f1-49db-8972-b5b9567dee77, status=NonOperational, nonOperationalReason=NETWORK_UNREACHABLE, stopSpmFailureLogged=false), log id: 49285c86 >2014-09-09 16:55:01,339 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-72) [64385954] FINISH, SetVdsStatusVDSCommand, log id: 49285c86 >2014-09-09 16:55:01,346 ERROR [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-72) [64385954] Host rack is set to Non-Operational, it is missing the following networks: rhevm >2014-09-09 16:55:01,366 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-72) [64385954] Correlation ID: 64385954, Job ID: 03478996-4a4d-4ffa-b44f-bc2efc45b340, Call Stack: null, Custom Event ID: -1, Message: Host rack does not comply with the cluster Default networks, the following networks are missing on host: 'rhevm' >2014-09-09 16:55:01,391 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-72) [d9736a9] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 1fc56aac-07f1-49db-8972-b5b9567dee77 Type: VDS >2014-09-09 16:55:01,416 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-72) [d9736a9] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host rack was set to NonOperational. >2014-09-09 16:55:01,467 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-72) [3822e820] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 1fc56aac-07f1-49db-8972-b5b9567dee77 Type: VDS >2014-09-09 16:55:01,469 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-72) [3822e820] Host 1fc56aac-07f1-49db-8972-b5b9567dee77 : rack is already in NonOperational status for reason NETWORK_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-09-09 17:00:00,024 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-51) Autorecovering 1 hosts >2014-09-09 17:00:00,024 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-51) Autorecovering hosts id: 1fc56aac-07f1-49db-8972-b5b9567dee77, name : rack >2014-09-09 17:00:00,030 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-51) [5c5dd985] Lock Acquired to object EngineLock [exclusiveLocks= key: 1fc56aac-07f1-49db-8972-b5b9567dee77 value: VDS >, sharedLocks= ] >2014-09-09 17:00:00,030 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-51) [5c5dd985] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 1fc56aac-07f1-49db-8972-b5b9567dee77 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2014-09-09 17:00:00,030 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-51) [5c5dd985] Before acquiring lock in order to prevent monitoring for host rack from data-center Default >2014-09-09 17:00:00,030 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-51) [5c5dd985] Lock acquired, from now a monitoring of host will be skipped for host rack from data-center Default >2014-09-09 17:00:00,038 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-51) [5c5dd985] START, SetVdsStatusVDSCommand(HostName = rack, HostId = 1fc56aac-07f1-49db-8972-b5b9567dee77, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: b03071 >2014-09-09 17:00:00,063 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-51) [5c5dd985] FINISH, SetVdsStatusVDSCommand, log id: b03071 >2014-09-09 17:00:00,076 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-51) [5c5dd985] Activate finished. Lock released. Monitoring can run now for host rack from data-center Default >2014-09-09 17:00:00,079 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-51) [5c5dd985] Lock freed to object EngineLock [exclusiveLocks= key: 1fc56aac-07f1-49db-8972-b5b9567dee77 value: VDS >, sharedLocks= ] >2014-09-09 17:00:01,989 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-53) START, GetHardwareInfoVDSCommand(HostName = rack, HostId = 1fc56aac-07f1-49db-8972-b5b9567dee77, vds=Host[rack,1fc56aac-07f1-49db-8972-b5b9567dee77]), log id: 1675d3ae >2014-09-09 17:00:01,995 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-53) FINISH, GetHardwareInfoVDSCommand, log id: 1675d3ae >2014-09-09 17:00:02,109 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-53) [41fda303] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 1fc56aac-07f1-49db-8972-b5b9567dee77 Type: VDS >2014-09-09 17:00:02,113 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-53) [41fda303] START, SetVdsStatusVDSCommand(HostName = rack, HostId = 1fc56aac-07f1-49db-8972-b5b9567dee77, status=NonOperational, nonOperationalReason=NETWORK_UNREACHABLE, stopSpmFailureLogged=false), log id: 563f2d47 >2014-09-09 17:00:02,148 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-53) [41fda303] FINISH, SetVdsStatusVDSCommand, log id: 563f2d47 >2014-09-09 17:00:02,154 ERROR [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-53) [41fda303] Host rack is set to Non-Operational, it is missing the following networks: rhevm >2014-09-09 17:00:02,176 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-53) [41fda303] Correlation ID: 41fda303, Job ID: 10561b87-ee9c-4f13-b377-485a2585805e, Call Stack: null, Custom Event ID: -1, Message: Host rack does not comply with the cluster Default networks, the following networks are missing on host: 'rhevm' >2014-09-09 17:00:02,197 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-53) [db8b1b9] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 1fc56aac-07f1-49db-8972-b5b9567dee77 Type: VDS >2014-09-09 17:00:02,219 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-53) [db8b1b9] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host rack was set to NonOperational. >2014-09-09 17:00:02,283 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-53) [5733dd3e] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 1fc56aac-07f1-49db-8972-b5b9567dee77 Type: VDS >2014-09-09 17:00:02,286 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-53) [5733dd3e] Host 1fc56aac-07f1-49db-8972-b5b9567dee77 : rack is already in NonOperational status for reason NETWORK_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-09-09 17:05:00,036 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-30) Autorecovering 1 hosts >2014-09-09 17:05:00,036 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-30) Autorecovering hosts id: 1fc56aac-07f1-49db-8972-b5b9567dee77, name : rack >2014-09-09 17:05:00,042 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-30) [5b453678] Lock Acquired to object EngineLock [exclusiveLocks= key: 1fc56aac-07f1-49db-8972-b5b9567dee77 value: VDS >, sharedLocks= ] >2014-09-09 17:05:00,042 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-30) [5b453678] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 1fc56aac-07f1-49db-8972-b5b9567dee77 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2014-09-09 17:05:00,042 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-30) [5b453678] Before acquiring lock in order to prevent monitoring for host rack from data-center Default >2014-09-09 17:05:00,042 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-30) [5b453678] Lock acquired, from now a monitoring of host will be skipped for host rack from data-center Default >2014-09-09 17:05:00,052 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-30) [5b453678] START, SetVdsStatusVDSCommand(HostName = rack, HostId = 1fc56aac-07f1-49db-8972-b5b9567dee77, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 6556720e >2014-09-09 17:05:00,077 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-30) [5b453678] FINISH, SetVdsStatusVDSCommand, log id: 6556720e >2014-09-09 17:05:00,090 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-30) [5b453678] Activate finished. Lock released. Monitoring can run now for host rack from data-center Default >2014-09-09 17:05:00,093 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-30) [5b453678] Lock freed to object EngineLock [exclusiveLocks= key: 1fc56aac-07f1-49db-8972-b5b9567dee77 value: VDS >, sharedLocks= ] >2014-09-09 17:05:02,793 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-35) START, GetHardwareInfoVDSCommand(HostName = rack, HostId = 1fc56aac-07f1-49db-8972-b5b9567dee77, vds=Host[rack,1fc56aac-07f1-49db-8972-b5b9567dee77]), log id: 42eb5942 >2014-09-09 17:05:02,798 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-35) FINISH, GetHardwareInfoVDSCommand, log id: 42eb5942 >2014-09-09 17:05:02,888 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-35) [2ee76c6a] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 1fc56aac-07f1-49db-8972-b5b9567dee77 Type: VDS >2014-09-09 17:05:02,891 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-35) [2ee76c6a] START, SetVdsStatusVDSCommand(HostName = rack, HostId = 1fc56aac-07f1-49db-8972-b5b9567dee77, status=NonOperational, nonOperationalReason=NETWORK_UNREACHABLE, stopSpmFailureLogged=false), log id: 6dbb8460 >2014-09-09 17:05:02,917 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-35) [2ee76c6a] FINISH, SetVdsStatusVDSCommand, log id: 6dbb8460 >2014-09-09 17:05:02,923 ERROR [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-35) [2ee76c6a] Host rack is set to Non-Operational, it is missing the following networks: rhevm >2014-09-09 17:05:02,945 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-35) [2ee76c6a] Correlation ID: 2ee76c6a, Job ID: 54b383cb-fe4a-4faf-a4d5-53f34a6fab1b, Call Stack: null, Custom Event ID: -1, Message: Host rack does not comply with the cluster Default networks, the following networks are missing on host: 'rhevm' >2014-09-09 17:05:02,974 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-35) [38973ef1] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 1fc56aac-07f1-49db-8972-b5b9567dee77 Type: VDS >2014-09-09 17:05:03,006 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-35) [38973ef1] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host rack was set to NonOperational. >2014-09-09 17:05:03,056 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-35) [4d22a9e0] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 1fc56aac-07f1-49db-8972-b5b9567dee77 Type: VDS >2014-09-09 17:05:03,059 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-35) [4d22a9e0] Host 1fc56aac-07f1-49db-8972-b5b9567dee77 : rack is already in NonOperational status for reason NETWORK_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-09-09 17:10:00,030 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-8) [28908be8] Autorecovering 1 hosts >2014-09-09 17:10:00,030 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-8) [28908be8] Autorecovering hosts id: 1fc56aac-07f1-49db-8972-b5b9567dee77, name : rack >2014-09-09 17:10:00,035 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-8) [13036fef] Lock Acquired to object EngineLock [exclusiveLocks= key: 1fc56aac-07f1-49db-8972-b5b9567dee77 value: VDS >, sharedLocks= ] >2014-09-09 17:10:00,035 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-8) [13036fef] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 1fc56aac-07f1-49db-8972-b5b9567dee77 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2014-09-09 17:10:00,036 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-8) [13036fef] Before acquiring lock in order to prevent monitoring for host rack from data-center Default >2014-09-09 17:10:00,036 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-8) [13036fef] Lock acquired, from now a monitoring of host will be skipped for host rack from data-center Default >2014-09-09 17:10:00,045 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-8) [13036fef] START, SetVdsStatusVDSCommand(HostName = rack, HostId = 1fc56aac-07f1-49db-8972-b5b9567dee77, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 535fb0a6 >2014-09-09 17:10:00,072 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-8) [13036fef] FINISH, SetVdsStatusVDSCommand, log id: 535fb0a6 >2014-09-09 17:10:00,085 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-8) [13036fef] Activate finished. Lock released. Monitoring can run now for host rack from data-center Default >2014-09-09 17:10:00,088 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-8) [13036fef] Lock freed to object EngineLock [exclusiveLocks= key: 1fc56aac-07f1-49db-8972-b5b9567dee77 value: VDS >, sharedLocks= ] >2014-09-09 17:10:00,510 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to dell-r210ii-16.rhev.lab.eng.brq.redhat.com/10.34.62.208 >2014-09-09 17:11:06,107 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection timed out > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:11:06,108 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection timed out > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:11:06,109 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-9) Exception during connection: java.util.concurrent.ExecutionException: java.net.ConnectException: Connection timed out > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:98) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:56) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:57) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:175) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection timed out > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:11:06,124 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-9) Command GetCapabilitiesVDSCommand(HostName = rack, HostId = 1fc56aac-07f1-49db-8972-b5b9567dee77, vds=Host[rack,1fc56aac-07f1-49db-8972-b5b9567dee77]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection timed out >2014-09-09 17:11:06,127 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-9) Host rack is not responding. It will stay in Connecting state for a grace period of 60 seconds and after that an attempt to fence the host will be issued. >2014-09-09 17:11:06,144 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-9) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host rack is not responding. It will stay in Connecting state for a grace period of 60 seconds and after that an attempt to fence the host will be issued. >2014-09-09 17:11:06,177 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-9) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection timed out > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:11:06,184 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-9) Failed to refresh VDS , vds = 1fc56aac-07f1-49db-8972-b5b9567dee77 : rack, VDS Network Error, continuing. >java.net.ConnectException: Connection timed out >2014-09-09 17:11:09,194 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to dell-r210ii-16.rhev.lab.eng.brq.redhat.com/10.34.62.208 >2014-09-09 17:11:12,195 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:11:12,196 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:11:12,196 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-36) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:98) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:56) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:57) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:175) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:11:12,203 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-36) Command GetCapabilitiesVDSCommand(HostName = rack, HostId = 1fc56aac-07f1-49db-8972-b5b9567dee77, vds=Host[rack,1fc56aac-07f1-49db-8972-b5b9567dee77]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-09-09 17:11:12,204 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-36) Host rack is not responding. It will stay in Connecting state for a grace period of 60 seconds and after that an attempt to fence the host will be issued. >2014-09-09 17:11:12,210 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-36) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:11:12,227 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-36) Failed to refresh VDS , vds = 1fc56aac-07f1-49db-8972-b5b9567dee77 : rack, VDS Network Error, continuing. >java.net.NoRouteToHostException: No route to host >2014-09-09 17:11:15,235 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to dell-r210ii-16.rhev.lab.eng.brq.redhat.com/10.34.62.208 >2014-09-09 17:11:18,237 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:11:18,238 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:11:18,239 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-51) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:98) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:56) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:57) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:175) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:11:18,251 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-51) [5c5dd985] Command GetCapabilitiesVDSCommand(HostName = rack, HostId = 1fc56aac-07f1-49db-8972-b5b9567dee77, vds=Host[rack,1fc56aac-07f1-49db-8972-b5b9567dee77]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-09-09 17:11:18,251 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-51) [5c5dd985] Host rack is not responding. It will stay in Connecting state for a grace period of 60 seconds and after that an attempt to fence the host will be issued. >2014-09-09 17:11:18,314 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-51) [5c5dd985] Server failed to respond, vds_id = 1fc56aac-07f1-49db-8972-b5b9567dee77, vds_name = rack, vm_count = 0, spm_status = None, non-responsive_timeout (seconds) = 60, error = java.net.NoRouteToHostException: No route to host >2014-09-09 17:11:18,328 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-51) [5c5dd985] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) > at sun.nio.ch.Net.connect(Net.java:465) > at sun.nio.ch.Net.connect(Net.java:457) > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) >, Custom Event ID: -1, Message: Host rack is non responsive. >2014-09-09 17:11:18,334 INFO [org.ovirt.engine.core.bll.VdsEventListener] (org.ovirt.thread.pool-6-thread-38) [5c5dd985] ResourceManager::vdsNotResponding entered for Host 1fc56aac-07f1-49db-8972-b5b9567dee77, dell-r210ii-16.rhev.lab.eng.brq.redhat.com >2014-09-09 17:11:18,371 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-51) [5c5dd985] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:11:18,422 INFO [org.ovirt.engine.core.bll.SshSoftFencingCommand] (org.ovirt.thread.pool-6-thread-38) [3c358b01] Running command: SshSoftFencingCommand internal: true. Entities affected : ID: 1fc56aac-07f1-49db-8972-b5b9567dee77 Type: VDS >2014-09-09 17:11:18,426 INFO [org.ovirt.engine.core.bll.SshSoftFencingCommand] (org.ovirt.thread.pool-6-thread-38) [3c358b01] Opening SSH Soft Fencing session on host dell-r210ii-16.rhev.lab.eng.brq.redhat.com >2014-09-09 17:11:21,240 ERROR [org.ovirt.engine.core.bll.SshSoftFencingCommand] (org.ovirt.thread.pool-6-thread-38) [3c358b01] SSH Soft Fencing command failed on host dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Failed to get the session. >Stdout: >Stderr: >Stacktrace: org.apache.sshd.common.RuntimeSshException: Failed to get the session.: org.apache.sshd.common.RuntimeSshException: Failed to get the session. > at org.apache.sshd.client.future.DefaultConnectFuture.getSession(DefaultConnectFuture.java:43) > at org.ovirt.engine.core.uutils.ssh.SSHClient.connect(SSHClient.java:294) [uutils.jar:] > at org.ovirt.engine.core.bll.utils.EngineSSHClient.connect(EngineSSHClient.java:59) [bll.jar:] > at org.ovirt.engine.core.bll.SshSoftFencingCommand.executeSshSoftFencingCommand(SshSoftFencingCommand.java:87) [bll.jar:] > at org.ovirt.engine.core.bll.SshSoftFencingCommand.executeCommand(SshSoftFencingCommand.java:53) [bll.jar:] > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1167) [bll.jar:] > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1298) [bll.jar:] > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1920) [bll.jar:] > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) [utils.jar:] > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) [utils.jar:] > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1319) [bll.jar:] > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:344) [bll.jar:] > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:430) [bll.jar:] > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:411) [bll.jar:] > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:616) [bll.jar:] > at sun.reflect.GeneratedMethodAccessor78.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptor.java:52) [jboss-as-ee.jar:7.4.0.Final-redhat-19] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:63) [jboss-as-ee.jar:7.4.0.Final-redhat-19] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:374) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.as.weld.ejb.Jsr299BindingsInterceptor.delegateInterception(Jsr299BindingsInterceptor.java:74) [jboss-as-weld.jar:7.4.0.Final-redhat-19] > at org.jboss.as.weld.ejb.Jsr299BindingsInterceptor.doMethodInterception(Jsr299BindingsInterceptor.java:84) [jboss-as-weld.jar:7.4.0.Final-redhat-19] > at org.jboss.as.weld.ejb.Jsr299BindingsInterceptor.processInvocation(Jsr299BindingsInterceptor.java:97) [jboss-as-weld.jar:7.4.0.Final-redhat-19] > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:63) [jboss-as-ee.jar:7.4.0.Final-redhat-19] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:63) [jboss-as-ee.jar:7.4.0.Final-redhat-19] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) [jboss-as-ejb3.jar:7.4.0.Final-redhat-19] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.as.weld.ejb.EjbRequestScopeActivationInterceptor.processInvocation(EjbRequestScopeActivationInterceptor.java:93) [jboss-as-weld.jar:7.4.0.Final-redhat-19] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) [jboss-as-ee.jar:7.4.0.Final-redhat-19] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) [jboss-as-ejb3.jar:7.4.0.Final-redhat-19] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) [jboss-as-ejb3.jar:7.4.0.Final-redhat-19] > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) [jboss-as-ejb3.jar:7.4.0.Final-redhat-19] > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) [jboss-as-ejb3.jar:7.4.0.Final-redhat-19] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) [jboss-as-ejb3.jar:7.4.0.Final-redhat-19] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) [jboss-as-ejb3.jar:7.4.0.Final-redhat-19] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) [jboss-as-ejb3.jar:7.4.0.Final-redhat-19] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) [jboss-as-ee.jar:7.4.0.Final-redhat-19] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) [jboss-as-ee.jar:7.4.0.Final-redhat-19] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:185) [jboss-as-ee.jar:7.4.0.Final-redhat-19] > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) [jboss-as-ee.jar:7.4.0.Final-redhat-19] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:73) [jboss-as-ee.jar:7.4.0.Final-redhat-19] > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) [bll.jar:] > at org.ovirt.engine.core.bll.VdsEventListener$4.run(VdsEventListener.java:243) [bll.jar:] > at org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil$InternalWrapperRunnable.run(ThreadPoolUtil.java:90) [utils.jar:] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_65] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_65] > at java.lang.Thread.run(Thread.java:745) [rt.jar:1.7.0_65] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.UnixAsynchronousSocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.UnixAsynchronousSocketChannelImpl.finishConnect(UnixAsynchronousSocketChannelImpl.java:252) [rt.jar:1.7.0_65] > at sun.nio.ch.UnixAsynchronousSocketChannelImpl.finish(UnixAsynchronousSocketChannelImpl.java:198) [rt.jar:1.7.0_65] > at sun.nio.ch.UnixAsynchronousSocketChannelImpl.onEvent(UnixAsynchronousSocketChannelImpl.java:213) [rt.jar:1.7.0_65] > at sun.nio.ch.EPollPort$EventHandlerTask.run(EPollPort.java:293) [rt.jar:1.7.0_65] > ... 1 more > >2014-09-09 17:11:21,309 INFO [org.ovirt.engine.core.bll.VdsNotRespondingTreatmentCommand] (org.ovirt.thread.pool-6-thread-38) [483ca656] Lock Acquired to object EngineLock [exclusiveLocks= key: 1fc56aac-07f1-49db-8972-b5b9567dee77 value: VDS_FENCE >, sharedLocks= ] >2014-09-09 17:11:21,309 ERROR [org.ovirt.engine.core.bll.VdsNotRespondingTreatmentCommand] (org.ovirt.thread.pool-6-thread-38) [483ca656] Failed to run Fence script on vds:rack. >2014-09-09 17:11:21,326 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-38) [483ca656] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host rack became non responsive. It has no power management configured. Please check the host status, manually reboot it, and click "Confirm Host Has Been Rebooted" >2014-09-09 17:11:21,326 WARN [org.ovirt.engine.core.bll.VdsNotRespondingTreatmentCommand] (org.ovirt.thread.pool-6-thread-38) [483ca656] CanDoAction of action VdsNotRespondingTreatment failed. Reasons:VAR__ACTION__RESTART,VDS_FENCE_DISABLED >2014-09-09 17:11:21,327 INFO [org.ovirt.engine.core.bll.VdsNotRespondingTreatmentCommand] (org.ovirt.thread.pool-6-thread-38) [483ca656] Lock freed to object EngineLock [exclusiveLocks= key: 1fc56aac-07f1-49db-8972-b5b9567dee77 value: VDS_FENCE >, sharedLocks= ] >2014-09-09 17:11:21,391 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to dell-r210ii-16.rhev.lab.eng.brq.redhat.com/10.34.62.208 >2014-09-09 17:11:24,239 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:11:24,240 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:11:24,240 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-55) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:98) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:56) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:57) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:175) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:11:24,265 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-55) Command GetCapabilitiesVDSCommand(HostName = rack, HostId = 1fc56aac-07f1-49db-8972-b5b9567dee77, vds=Host[rack,1fc56aac-07f1-49db-8972-b5b9567dee77]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-09-09 17:11:24,265 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-55) Host rack is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-09 17:11:24,310 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-55) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:11:27,336 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to dell-r210ii-16.rhev.lab.eng.brq.redhat.com/10.34.62.208 >2014-09-09 17:11:30,337 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:11:30,337 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:11:30,338 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-69) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:98) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:56) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:57) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:175) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:11:30,342 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-69) [2f5064a8] Command GetCapabilitiesVDSCommand(HostName = rack, HostId = 1fc56aac-07f1-49db-8972-b5b9567dee77, vds=Host[rack,1fc56aac-07f1-49db-8972-b5b9567dee77]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-09-09 17:11:30,342 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-69) [2f5064a8] Host rack is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-09 17:11:30,373 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-69) [2f5064a8] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:11:33,390 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to dell-r210ii-16.rhev.lab.eng.brq.redhat.com/10.34.62.208 >2014-09-09 17:11:36,392 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:11:36,393 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:11:36,393 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-84) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:98) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:56) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:57) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:175) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:11:36,396 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-84) Command GetCapabilitiesVDSCommand(HostName = rack, HostId = 1fc56aac-07f1-49db-8972-b5b9567dee77, vds=Host[rack,1fc56aac-07f1-49db-8972-b5b9567dee77]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-09-09 17:11:36,396 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-84) Host rack is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-09 17:11:36,428 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-84) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:11:39,444 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to dell-r210ii-16.rhev.lab.eng.brq.redhat.com/10.34.62.208 >2014-09-09 17:11:42,446 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:11:42,446 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:11:42,447 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-92) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:98) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:56) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:57) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:175) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:11:42,449 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-92) Command GetCapabilitiesVDSCommand(HostName = rack, HostId = 1fc56aac-07f1-49db-8972-b5b9567dee77, vds=Host[rack,1fc56aac-07f1-49db-8972-b5b9567dee77]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-09-09 17:11:42,449 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-92) Host rack is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-09 17:11:42,480 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-92) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:11:45,495 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to dell-r210ii-16.rhev.lab.eng.brq.redhat.com/10.34.62.208 >2014-09-09 17:11:48,496 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:11:48,497 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:11:48,498 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-11) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:98) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:56) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:57) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:175) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:11:48,503 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-11) Command GetCapabilitiesVDSCommand(HostName = rack, HostId = 1fc56aac-07f1-49db-8972-b5b9567dee77, vds=Host[rack,1fc56aac-07f1-49db-8972-b5b9567dee77]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-09-09 17:11:48,503 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-11) Host rack is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-09 17:11:48,528 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-11) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:11:51,543 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to dell-r210ii-16.rhev.lab.eng.brq.redhat.com/10.34.62.208 >2014-09-09 17:11:54,545 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:11:54,545 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:11:54,546 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-17) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:98) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:56) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:57) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:175) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:11:54,549 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-17) Command GetCapabilitiesVDSCommand(HostName = rack, HostId = 1fc56aac-07f1-49db-8972-b5b9567dee77, vds=Host[rack,1fc56aac-07f1-49db-8972-b5b9567dee77]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-09-09 17:11:54,550 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-17) Host rack is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-09 17:11:54,582 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-17) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:11:57,606 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to dell-r210ii-16.rhev.lab.eng.brq.redhat.com/10.34.62.208 >2014-09-09 17:12:07,608 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:12:07,608 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:12:07,609 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-34) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:98) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:56) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:57) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:175) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:12:07,611 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-34) Command GetCapabilitiesVDSCommand(HostName = rack, HostId = 1fc56aac-07f1-49db-8972-b5b9567dee77, vds=Host[rack,1fc56aac-07f1-49db-8972-b5b9567dee77]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-09-09 17:12:07,611 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-34) Host rack is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-09 17:12:07,641 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-34) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:12:10,657 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to dell-r210ii-16.rhev.lab.eng.brq.redhat.com/10.34.62.208 >2014-09-09 17:12:13,659 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:12:13,659 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:12:13,660 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-59) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:98) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:56) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:57) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:175) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:12:13,663 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-59) Command GetCapabilitiesVDSCommand(HostName = rack, HostId = 1fc56aac-07f1-49db-8972-b5b9567dee77, vds=Host[rack,1fc56aac-07f1-49db-8972-b5b9567dee77]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-09-09 17:12:13,663 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-59) Host rack is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-09 17:12:13,696 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-59) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:12:16,714 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to dell-r210ii-16.rhev.lab.eng.brq.redhat.com/10.34.62.208 >2014-09-09 17:12:19,716 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:12:19,716 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:12:19,717 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-70) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:98) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:56) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:57) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:175) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:12:19,719 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-70) Command GetCapabilitiesVDSCommand(HostName = rack, HostId = 1fc56aac-07f1-49db-8972-b5b9567dee77, vds=Host[rack,1fc56aac-07f1-49db-8972-b5b9567dee77]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-09-09 17:12:19,720 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-70) Host rack is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-09 17:12:19,750 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-70) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:12:22,764 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to dell-r210ii-16.rhev.lab.eng.brq.redhat.com/10.34.62.208 >2014-09-09 17:12:25,765 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:12:25,765 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:12:25,766 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-86) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:98) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:56) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:57) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:175) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:12:25,768 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-86) Command GetCapabilitiesVDSCommand(HostName = rack, HostId = 1fc56aac-07f1-49db-8972-b5b9567dee77, vds=Host[rack,1fc56aac-07f1-49db-8972-b5b9567dee77]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-09-09 17:12:25,768 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-86) Host rack is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-09 17:12:25,793 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-86) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:12:28,809 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to dell-r210ii-16.rhev.lab.eng.brq.redhat.com/10.34.62.208 >2014-09-09 17:12:31,811 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:12:31,811 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:12:31,812 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-95) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:98) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:56) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:57) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:175) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:12:31,814 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-95) [24a2c919] Command GetCapabilitiesVDSCommand(HostName = rack, HostId = 1fc56aac-07f1-49db-8972-b5b9567dee77, vds=Host[rack,1fc56aac-07f1-49db-8972-b5b9567dee77]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-09-09 17:12:31,814 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-95) [24a2c919] Host rack is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-09 17:12:31,841 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-95) [24a2c919] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:12:34,859 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to dell-r210ii-16.rhev.lab.eng.brq.redhat.com/10.34.62.208 >2014-09-09 17:12:37,860 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:12:37,860 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:12:37,861 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-11) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:98) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:56) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:57) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:175) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:12:37,864 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-11) Command GetCapabilitiesVDSCommand(HostName = rack, HostId = 1fc56aac-07f1-49db-8972-b5b9567dee77, vds=Host[rack,1fc56aac-07f1-49db-8972-b5b9567dee77]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-09-09 17:12:37,864 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-11) Host rack is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-09 17:12:37,894 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-11) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:12:40,921 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to dell-r210ii-16.rhev.lab.eng.brq.redhat.com/10.34.62.208 >2014-09-09 17:12:43,922 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:12:43,922 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:12:43,923 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-25) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:98) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:56) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:57) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:175) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:12:43,926 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-25) [27472998] Command GetCapabilitiesVDSCommand(HostName = rack, HostId = 1fc56aac-07f1-49db-8972-b5b9567dee77, vds=Host[rack,1fc56aac-07f1-49db-8972-b5b9567dee77]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-09-09 17:12:43,926 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-25) [27472998] Host rack is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-09 17:12:43,959 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-25) [27472998] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:12:46,976 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to dell-r210ii-16.rhev.lab.eng.brq.redhat.com/10.34.62.208 >2014-09-09 17:12:49,978 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:12:49,978 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:12:49,979 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-32) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:98) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:56) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:57) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:175) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:12:49,981 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-32) Command GetCapabilitiesVDSCommand(HostName = rack, HostId = 1fc56aac-07f1-49db-8972-b5b9567dee77, vds=Host[rack,1fc56aac-07f1-49db-8972-b5b9567dee77]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-09-09 17:12:49,982 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-32) Host rack is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-09 17:12:50,017 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-32) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:12:53,035 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to dell-r210ii-16.rhev.lab.eng.brq.redhat.com/10.34.62.208 >2014-09-09 17:12:56,036 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:12:56,036 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:12:56,037 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-50) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:98) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:56) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:57) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:175) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:12:56,039 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-50) Command GetCapabilitiesVDSCommand(HostName = rack, HostId = 1fc56aac-07f1-49db-8972-b5b9567dee77, vds=Host[rack,1fc56aac-07f1-49db-8972-b5b9567dee77]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-09-09 17:12:56,040 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-50) Host rack is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-09 17:12:56,070 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-50) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:12:59,084 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to dell-r210ii-16.rhev.lab.eng.brq.redhat.com/10.34.62.208 >2014-09-09 17:13:02,086 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:13:02,086 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:13:02,087 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-59) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:98) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:56) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:57) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:175) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:13:02,089 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-59) Command GetCapabilitiesVDSCommand(HostName = rack, HostId = 1fc56aac-07f1-49db-8972-b5b9567dee77, vds=Host[rack,1fc56aac-07f1-49db-8972-b5b9567dee77]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-09-09 17:13:02,090 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-59) Host rack is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-09 17:13:02,117 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-59) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:13:05,133 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to dell-r210ii-16.rhev.lab.eng.brq.redhat.com/10.34.62.208 >2014-09-09 17:13:08,135 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:13:08,135 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:13:08,136 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-70) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:98) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:56) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:57) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:175) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:13:08,140 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-70) Command GetCapabilitiesVDSCommand(HostName = rack, HostId = 1fc56aac-07f1-49db-8972-b5b9567dee77, vds=Host[rack,1fc56aac-07f1-49db-8972-b5b9567dee77]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-09-09 17:13:08,140 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-70) Host rack is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-09 17:13:08,172 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-70) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:13:11,187 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to dell-r210ii-16.rhev.lab.eng.brq.redhat.com/10.34.62.208 >2014-09-09 17:13:14,188 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:13:14,189 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:13:14,189 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-87) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:98) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:56) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:57) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:175) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:13:14,194 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-87) [53e7c8ba] Command GetCapabilitiesVDSCommand(HostName = rack, HostId = 1fc56aac-07f1-49db-8972-b5b9567dee77, vds=Host[rack,1fc56aac-07f1-49db-8972-b5b9567dee77]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-09-09 17:13:14,195 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-87) [53e7c8ba] Host rack is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-09 17:13:14,227 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-87) [53e7c8ba] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:13:17,242 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to dell-r210ii-16.rhev.lab.eng.brq.redhat.com/10.34.62.208 >2014-09-09 17:13:17,243 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:13:17,244 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:13:17,244 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-3) Exception during connection: java.util.concurrent.ExecutionException: java.net.ConnectException: Connection refused > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:98) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:56) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:57) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:175) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:13:17,246 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-3) Command GetCapabilitiesVDSCommand(HostName = rack, HostId = 1fc56aac-07f1-49db-8972-b5b9567dee77, vds=Host[rack,1fc56aac-07f1-49db-8972-b5b9567dee77]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-09-09 17:13:17,246 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-3) Host rack is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-09 17:13:17,275 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-3) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:13:20,291 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to dell-r210ii-16.rhev.lab.eng.brq.redhat.com/10.34.62.208 >2014-09-09 17:13:20,292 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:13:20,292 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:13:20,293 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-14) Exception during connection: java.util.concurrent.ExecutionException: java.net.ConnectException: Connection refused > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:98) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:56) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:57) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:175) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:13:20,296 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-14) Command GetCapabilitiesVDSCommand(HostName = rack, HostId = 1fc56aac-07f1-49db-8972-b5b9567dee77, vds=Host[rack,1fc56aac-07f1-49db-8972-b5b9567dee77]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-09-09 17:13:20,296 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-14) Host rack is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-09 17:13:20,333 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-14) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:13:23,359 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to dell-r210ii-16.rhev.lab.eng.brq.redhat.com/10.34.62.208 >2014-09-09 17:13:30,361 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:13:30,361 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:13:30,361 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-19) Exception during connection: java.util.concurrent.ExecutionException: java.net.ConnectException: Connection refused > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:98) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:56) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:57) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:175) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:13:30,365 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-19) Command GetCapabilitiesVDSCommand(HostName = rack, HostId = 1fc56aac-07f1-49db-8972-b5b9567dee77, vds=Host[rack,1fc56aac-07f1-49db-8972-b5b9567dee77]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-09-09 17:13:30,365 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-19) Host rack is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-09 17:13:30,397 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-19) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:13:33,414 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to dell-r210ii-16.rhev.lab.eng.brq.redhat.com/10.34.62.208 >2014-09-09 17:13:33,415 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.Reactor] (SSL Stomp Reactor) Unable to process messages: java.nio.channels.ClosedChannelException > at sun.nio.ch.SocketChannelImpl.ensureReadOpen(SocketChannelImpl.java:252) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:295) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.SSLEngineNioHelper.read(SSLEngineNioHelper.java:51) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.SSLClient.read(SSLClient.java:81) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.readBuffer(ReactorClient.java:203) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.stomp.StompCommonClient.processIncoming(StompCommonClient.java:90) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.process(ReactorClient.java:144) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.SSLClient.process(SSLClient.java:115) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.processChannels(Reactor.java:86) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:62) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:16:35,608 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-36) Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@7b9db3b4 >2014-09-09 17:16:35,608 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-36) HostName = rack >2014-09-09 17:16:35,608 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-36) Command GetCapabilitiesVDSCommand(HostName = rack, HostId = 1fc56aac-07f1-49db-8972-b5b9567dee77, vds=Host[rack,1fc56aac-07f1-49db-8972-b5b9567dee77]) execution failed. Exception: VDSNetworkException: VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues' >2014-09-09 17:16:35,608 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-36) Host rack is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-09 17:16:35,642 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-36) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host rack is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-09 17:16:35,676 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-36) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues' > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:183) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:16) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] > >2014-09-09 17:16:38,698 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to dell-r210ii-16.rhev.lab.eng.brq.redhat.com/10.34.62.208 >2014-09-09 17:16:41,700 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:16:41,700 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:16:41,701 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-98) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:98) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:56) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:57) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:175) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:16:41,704 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-98) Command GetCapabilitiesVDSCommand(HostName = rack, HostId = 1fc56aac-07f1-49db-8972-b5b9567dee77, vds=Host[rack,1fc56aac-07f1-49db-8972-b5b9567dee77]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-09-09 17:16:41,704 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-98) Host rack is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-09 17:16:41,731 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-98) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:16:44,745 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to dell-r210ii-16.rhev.lab.eng.brq.redhat.com/10.34.62.208 >2014-09-09 17:16:47,747 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:16:47,748 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:16:47,748 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-14) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:98) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:56) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:57) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:175) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:16:47,752 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-14) Command GetCapabilitiesVDSCommand(HostName = rack, HostId = 1fc56aac-07f1-49db-8972-b5b9567dee77, vds=Host[rack,1fc56aac-07f1-49db-8972-b5b9567dee77]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-09-09 17:16:47,753 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-14) Host rack is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-09 17:16:47,788 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-14) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:16:50,807 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to dell-r210ii-16.rhev.lab.eng.brq.redhat.com/10.34.62.208 >2014-09-09 17:16:53,809 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:16:53,809 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:16:53,810 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-23) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:98) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:56) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:57) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:175) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:16:53,816 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-23) Command GetCapabilitiesVDSCommand(HostName = rack, HostId = 1fc56aac-07f1-49db-8972-b5b9567dee77, vds=Host[rack,1fc56aac-07f1-49db-8972-b5b9567dee77]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-09-09 17:16:53,816 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-23) Host rack is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-09 17:16:53,846 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-23) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:16:56,861 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to dell-r210ii-16.rhev.lab.eng.brq.redhat.com/10.34.62.208 >2014-09-09 17:17:14,862 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:17:14,862 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:17:14,863 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-40) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:98) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:56) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:57) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:175) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:17:14,865 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-40) Command GetCapabilitiesVDSCommand(HostName = rack, HostId = 1fc56aac-07f1-49db-8972-b5b9567dee77, vds=Host[rack,1fc56aac-07f1-49db-8972-b5b9567dee77]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-09-09 17:17:14,866 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-40) Host rack is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-09 17:17:14,919 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-40) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:17:17,949 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to dell-r210ii-16.rhev.lab.eng.brq.redhat.com/10.34.62.208 >2014-09-09 17:17:20,950 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:17:20,950 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:17:20,951 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-77) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:98) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:56) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:57) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:175) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:17:20,954 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-77) [1d4fb057] Command GetCapabilitiesVDSCommand(HostName = rack, HostId = 1fc56aac-07f1-49db-8972-b5b9567dee77, vds=Host[rack,1fc56aac-07f1-49db-8972-b5b9567dee77]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-09-09 17:17:20,955 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-77) [1d4fb057] Host rack is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-09 17:17:20,983 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-77) [1d4fb057] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:17:23,996 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to dell-r210ii-16.rhev.lab.eng.brq.redhat.com/10.34.62.208 >2014-09-09 17:17:26,997 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:17:26,998 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:17:26,998 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-91) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:98) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:56) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:57) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:175) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:17:27,001 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-91) Command GetCapabilitiesVDSCommand(HostName = rack, HostId = 1fc56aac-07f1-49db-8972-b5b9567dee77, vds=Host[rack,1fc56aac-07f1-49db-8972-b5b9567dee77]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-09-09 17:17:27,002 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-91) Host rack is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-09 17:17:27,031 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-91) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:17:30,051 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to dell-r210ii-16.rhev.lab.eng.brq.redhat.com/10.34.62.208 >2014-09-09 17:17:33,052 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:17:33,052 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:17:33,053 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-100) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:98) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:56) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:57) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:175) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:17:33,056 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-100) Command GetCapabilitiesVDSCommand(HostName = rack, HostId = 1fc56aac-07f1-49db-8972-b5b9567dee77, vds=Host[rack,1fc56aac-07f1-49db-8972-b5b9567dee77]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-09-09 17:17:33,057 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-100) Host rack is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-09 17:17:33,094 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-100) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:17:36,124 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to dell-r210ii-16.rhev.lab.eng.brq.redhat.com/10.34.62.208 >2014-09-09 17:17:39,125 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:17:39,125 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:17:39,126 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-18) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:98) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:56) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:57) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:175) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:17:39,128 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-18) Command GetCapabilitiesVDSCommand(HostName = rack, HostId = 1fc56aac-07f1-49db-8972-b5b9567dee77, vds=Host[rack,1fc56aac-07f1-49db-8972-b5b9567dee77]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-09-09 17:17:39,128 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-18) Host rack is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-09 17:17:39,156 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-18) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:17:42,177 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to dell-r210ii-16.rhev.lab.eng.brq.redhat.com/10.34.62.208 >2014-09-09 17:17:45,179 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:17:45,179 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:17:45,180 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-22) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:98) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:56) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:57) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:175) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:17:45,183 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-22) Command GetCapabilitiesVDSCommand(HostName = rack, HostId = 1fc56aac-07f1-49db-8972-b5b9567dee77, vds=Host[rack,1fc56aac-07f1-49db-8972-b5b9567dee77]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-09-09 17:17:45,184 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-22) Host rack is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-09 17:17:45,215 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-22) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:17:48,231 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to dell-r210ii-16.rhev.lab.eng.brq.redhat.com/10.34.62.208 >2014-09-09 17:17:51,232 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:17:51,232 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:17:51,233 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-38) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:98) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:56) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:57) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:175) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:17:51,235 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-38) Command GetCapabilitiesVDSCommand(HostName = rack, HostId = 1fc56aac-07f1-49db-8972-b5b9567dee77, vds=Host[rack,1fc56aac-07f1-49db-8972-b5b9567dee77]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-09-09 17:17:51,235 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-38) Host rack is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-09 17:17:51,262 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-38) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:17:54,275 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to dell-r210ii-16.rhev.lab.eng.brq.redhat.com/10.34.62.208 >2014-09-09 17:17:57,276 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:17:57,276 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:17:57,277 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-67) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:98) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:56) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:57) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:175) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:17:57,280 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-67) Command GetCapabilitiesVDSCommand(HostName = rack, HostId = 1fc56aac-07f1-49db-8972-b5b9567dee77, vds=Host[rack,1fc56aac-07f1-49db-8972-b5b9567dee77]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-09-09 17:17:57,280 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-67) Host rack is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-09 17:17:57,310 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-67) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:18:00,329 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to dell-r210ii-16.rhev.lab.eng.brq.redhat.com/10.34.62.208 >2014-09-09 17:18:03,330 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:18:03,330 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:18:03,331 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-71) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:98) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:56) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:57) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:175) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:18:03,339 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-71) Command GetCapabilitiesVDSCommand(HostName = rack, HostId = 1fc56aac-07f1-49db-8972-b5b9567dee77, vds=Host[rack,1fc56aac-07f1-49db-8972-b5b9567dee77]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-09-09 17:18:03,339 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-71) Host rack is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-09 17:18:03,367 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-71) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:18:06,385 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to dell-r210ii-16.rhev.lab.eng.brq.redhat.com/10.34.62.208 >2014-09-09 17:18:09,387 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:18:09,387 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:18:09,388 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-85) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:98) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:56) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:57) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:175) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:18:09,390 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-85) Command GetCapabilitiesVDSCommand(HostName = rack, HostId = 1fc56aac-07f1-49db-8972-b5b9567dee77, vds=Host[rack,1fc56aac-07f1-49db-8972-b5b9567dee77]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-09-09 17:18:09,391 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-85) Host rack is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-09 17:18:09,426 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-85) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:18:12,443 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to dell-r210ii-16.rhev.lab.eng.brq.redhat.com/10.34.62.208 >2014-09-09 17:18:13,445 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:18:13,446 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:18:13,446 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-86) Exception during connection: java.util.concurrent.ExecutionException: java.net.ConnectException: Connection refused > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:98) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:56) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:57) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:175) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:18:13,450 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-86) Command GetCapabilitiesVDSCommand(HostName = rack, HostId = 1fc56aac-07f1-49db-8972-b5b9567dee77, vds=Host[rack,1fc56aac-07f1-49db-8972-b5b9567dee77]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-09-09 17:18:13,450 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-86) Host rack is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-09 17:18:13,510 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-86) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:18:16,528 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to dell-r210ii-16.rhev.lab.eng.brq.redhat.com/10.34.62.208 >2014-09-09 17:18:16,529 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:18:16,529 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:18:16,530 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-95) Exception during connection: java.util.concurrent.ExecutionException: java.net.ConnectException: Connection refused > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:98) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:56) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:57) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:175) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:18:16,533 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-95) [24a2c919] Command GetCapabilitiesVDSCommand(HostName = rack, HostId = 1fc56aac-07f1-49db-8972-b5b9567dee77, vds=Host[rack,1fc56aac-07f1-49db-8972-b5b9567dee77]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-09-09 17:18:16,533 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-95) [24a2c919] Host rack is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-09 17:18:16,567 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-95) [24a2c919] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:18:19,583 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to dell-r210ii-16.rhev.lab.eng.brq.redhat.com/10.34.62.208 >2014-09-09 17:18:19,584 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:18:19,584 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:18:19,584 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-4) Exception during connection: java.util.concurrent.ExecutionException: java.net.ConnectException: Connection refused > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:98) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:56) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:57) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:175) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:18:19,587 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-4) Command GetCapabilitiesVDSCommand(HostName = rack, HostId = 1fc56aac-07f1-49db-8972-b5b9567dee77, vds=Host[rack,1fc56aac-07f1-49db-8972-b5b9567dee77]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-09-09 17:18:19,587 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-4) Host rack is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-09 17:18:19,624 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-4) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:18:22,639 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to dell-r210ii-16.rhev.lab.eng.brq.redhat.com/10.34.62.208 >2014-09-09 17:18:37,641 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.Reactor] (SSL Stomp Reactor) Unable to process messages: java.nio.channels.ClosedChannelException > at sun.nio.ch.SocketChannelImpl.ensureWriteOpen(SocketChannelImpl.java:265) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:474) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.SSLEngineNioHelper.write(SSLEngineNioHelper.java:104) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.SSLEngineNioHelper.process(SSLEngineNioHelper.java:125) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.SSLClient.pendingOperations(SSLClient.java:74) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.SSLClient.process(SSLClient.java:98) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.processChannels(Reactor.java:86) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:62) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:21:39,723 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-16) Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@7b0b2b6f >2014-09-09 17:21:39,723 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-16) HostName = rack >2014-09-09 17:21:39,723 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-16) Command GetCapabilitiesVDSCommand(HostName = rack, HostId = 1fc56aac-07f1-49db-8972-b5b9567dee77, vds=Host[rack,1fc56aac-07f1-49db-8972-b5b9567dee77]) execution failed. Exception: VDSNetworkException: VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues' >2014-09-09 17:21:39,723 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-16) Host rack is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-09 17:21:39,741 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-16) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host rack is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-09 17:21:39,767 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-16) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues' > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:183) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:16) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] > >2014-09-09 17:21:42,782 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to dell-r210ii-16.rhev.lab.eng.brq.redhat.com/10.34.62.208 >2014-09-09 17:21:57,784 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:21:57,784 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:21:57,784 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-2) Exception during connection: java.util.concurrent.ExecutionException: java.net.ConnectException: Connection refused > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:98) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:56) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:57) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:175) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:21:57,787 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-2) Command GetCapabilitiesVDSCommand(HostName = rack, HostId = 1fc56aac-07f1-49db-8972-b5b9567dee77, vds=Host[rack,1fc56aac-07f1-49db-8972-b5b9567dee77]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-09-09 17:21:57,787 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-2) Host rack is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-09 17:21:57,828 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-2) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:22:00,855 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to dell-r210ii-16.rhev.lab.eng.brq.redhat.com/10.34.62.208 >2014-09-09 17:22:03,856 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:22:03,856 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:22:03,857 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-47) Exception during connection: java.util.concurrent.ExecutionException: java.net.ConnectException: Connection refused > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:98) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:56) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:57) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:175) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:22:03,859 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-47) Command GetCapabilitiesVDSCommand(HostName = rack, HostId = 1fc56aac-07f1-49db-8972-b5b9567dee77, vds=Host[rack,1fc56aac-07f1-49db-8972-b5b9567dee77]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-09-09 17:22:03,860 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-47) Host rack is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-09 17:22:03,885 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-47) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:22:06,902 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to dell-r210ii-16.rhev.lab.eng.brq.redhat.com/10.34.62.208 >2014-09-09 17:22:06,903 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:22:06,904 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:22:06,904 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-57) Exception during connection: java.util.concurrent.ExecutionException: java.net.ConnectException: Connection refused > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:98) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:56) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:57) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:175) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:22:06,907 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-57) Command GetCapabilitiesVDSCommand(HostName = rack, HostId = 1fc56aac-07f1-49db-8972-b5b9567dee77, vds=Host[rack,1fc56aac-07f1-49db-8972-b5b9567dee77]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-09-09 17:22:06,907 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-57) Host rack is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-09 17:22:06,942 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-57) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:22:09,957 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to dell-r210ii-16.rhev.lab.eng.brq.redhat.com/10.34.62.208 >2014-09-09 17:22:09,957 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:22:09,958 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:22:09,958 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-67) Exception during connection: java.util.concurrent.ExecutionException: java.net.ConnectException: Connection refused > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:98) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:56) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:57) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:175) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:22:09,960 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-67) Command GetCapabilitiesVDSCommand(HostName = rack, HostId = 1fc56aac-07f1-49db-8972-b5b9567dee77, vds=Host[rack,1fc56aac-07f1-49db-8972-b5b9567dee77]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-09-09 17:22:09,961 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-67) Host rack is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-09 17:22:09,991 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-67) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:22:13,017 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to dell-r210ii-16.rhev.lab.eng.brq.redhat.com/10.34.62.208 >2014-09-09 17:22:13,018 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:22:13,019 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:22:13,019 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-59) Exception during connection: java.util.concurrent.ExecutionException: java.net.ConnectException: Connection refused > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:98) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:56) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:57) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:175) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:22:13,034 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-59) Command GetCapabilitiesVDSCommand(HostName = rack, HostId = 1fc56aac-07f1-49db-8972-b5b9567dee77, vds=Host[rack,1fc56aac-07f1-49db-8972-b5b9567dee77]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-09-09 17:22:13,034 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-59) Host rack is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-09 17:22:13,065 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-59) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:22:16,079 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to dell-r210ii-16.rhev.lab.eng.brq.redhat.com/10.34.62.208 >2014-09-09 17:22:16,080 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:22:16,080 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:22:16,081 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-71) Exception during connection: java.util.concurrent.ExecutionException: java.net.ConnectException: Connection refused > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:98) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:56) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:57) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:175) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:22:16,083 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-71) Command GetCapabilitiesVDSCommand(HostName = rack, HostId = 1fc56aac-07f1-49db-8972-b5b9567dee77, vds=Host[rack,1fc56aac-07f1-49db-8972-b5b9567dee77]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-09-09 17:22:16,083 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-71) Host rack is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-09 17:22:16,111 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-71) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:22:19,127 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to dell-r210ii-16.rhev.lab.eng.brq.redhat.com/10.34.62.208 >2014-09-09 17:22:19,127 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:22:19,128 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:22:19,128 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-75) Exception during connection: java.util.concurrent.ExecutionException: java.net.ConnectException: Connection refused > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:98) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:56) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:57) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:175) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:22:19,132 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-75) Command GetCapabilitiesVDSCommand(HostName = rack, HostId = 1fc56aac-07f1-49db-8972-b5b9567dee77, vds=Host[rack,1fc56aac-07f1-49db-8972-b5b9567dee77]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-09-09 17:22:19,132 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-75) Host rack is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-09 17:22:19,166 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-75) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:22:22,188 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to dell-r210ii-16.rhev.lab.eng.brq.redhat.com/10.34.62.208 >2014-09-09 17:22:22,189 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:22:22,189 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:22:22,189 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-90) Exception during connection: java.util.concurrent.ExecutionException: java.net.ConnectException: Connection refused > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:98) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:56) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:57) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:175) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:22:22,192 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-90) [18a5c631] Command GetCapabilitiesVDSCommand(HostName = rack, HostId = 1fc56aac-07f1-49db-8972-b5b9567dee77, vds=Host[rack,1fc56aac-07f1-49db-8972-b5b9567dee77]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-09-09 17:22:22,192 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-90) [18a5c631] Host rack is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-09 17:22:22,221 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-90) [18a5c631] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:22:25,239 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to dell-r210ii-16.rhev.lab.eng.brq.redhat.com/10.34.62.208 >2014-09-09 17:22:25,240 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:22:25,240 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:22:25,240 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-87) Exception during connection: java.util.concurrent.ExecutionException: java.net.ConnectException: Connection refused > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:98) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:56) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:57) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:175) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:22:25,244 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-87) [53e7c8ba] Command GetCapabilitiesVDSCommand(HostName = rack, HostId = 1fc56aac-07f1-49db-8972-b5b9567dee77, vds=Host[rack,1fc56aac-07f1-49db-8972-b5b9567dee77]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-09-09 17:22:25,244 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-87) [53e7c8ba] Host rack is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-09 17:22:25,282 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-87) [53e7c8ba] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:22:28,302 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to dell-r210ii-16.rhev.lab.eng.brq.redhat.com/10.34.62.208 >2014-09-09 17:22:28,302 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:22:28,303 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:22:28,303 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-98) Exception during connection: java.util.concurrent.ExecutionException: java.net.ConnectException: Connection refused > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:98) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:56) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:57) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:175) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:22:28,306 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-98) Command GetCapabilitiesVDSCommand(HostName = rack, HostId = 1fc56aac-07f1-49db-8972-b5b9567dee77, vds=Host[rack,1fc56aac-07f1-49db-8972-b5b9567dee77]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-09-09 17:22:28,307 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-98) Host rack is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-09 17:22:28,339 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-98) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:22:31,355 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to dell-r210ii-16.rhev.lab.eng.brq.redhat.com/10.34.62.208 >2014-09-09 17:22:31,356 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:22:31,357 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:22:31,357 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-12) Exception during connection: java.util.concurrent.ExecutionException: java.net.ConnectException: Connection refused > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:98) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:56) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:57) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:175) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:22:31,360 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-12) Command GetCapabilitiesVDSCommand(HostName = rack, HostId = 1fc56aac-07f1-49db-8972-b5b9567dee77, vds=Host[rack,1fc56aac-07f1-49db-8972-b5b9567dee77]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-09-09 17:22:31,360 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-12) Host rack is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-09 17:22:31,391 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-12) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:22:34,407 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to dell-r210ii-16.rhev.lab.eng.brq.redhat.com/10.34.62.208 >2014-09-09 17:22:34,407 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:22:34,408 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:22:34,409 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-18) Exception during connection: java.util.concurrent.ExecutionException: java.net.ConnectException: Connection refused > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:98) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:56) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:57) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:175) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:22:34,417 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-18) Command GetCapabilitiesVDSCommand(HostName = rack, HostId = 1fc56aac-07f1-49db-8972-b5b9567dee77, vds=Host[rack,1fc56aac-07f1-49db-8972-b5b9567dee77]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-09-09 17:22:34,417 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-18) Host rack is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-09 17:22:34,448 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-18) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:22:37,464 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to dell-r210ii-16.rhev.lab.eng.brq.redhat.com/10.34.62.208 >2014-09-09 17:22:37,465 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:22:37,465 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:22:37,466 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-24) Exception during connection: java.util.concurrent.ExecutionException: java.net.ConnectException: Connection refused > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:98) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:56) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:57) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:175) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:22:37,470 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-24) Command GetCapabilitiesVDSCommand(HostName = rack, HostId = 1fc56aac-07f1-49db-8972-b5b9567dee77, vds=Host[rack,1fc56aac-07f1-49db-8972-b5b9567dee77]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-09-09 17:22:37,470 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-24) Host rack is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-09 17:22:37,498 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-24) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:22:40,515 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to dell-r210ii-16.rhev.lab.eng.brq.redhat.com/10.34.62.208 >2014-09-09 17:22:40,516 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:22:40,517 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:22:40,517 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-19) Exception during connection: java.util.concurrent.ExecutionException: java.net.ConnectException: Connection refused > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:98) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:56) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:57) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:175) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:22:40,521 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-19) Command GetCapabilitiesVDSCommand(HostName = rack, HostId = 1fc56aac-07f1-49db-8972-b5b9567dee77, vds=Host[rack,1fc56aac-07f1-49db-8972-b5b9567dee77]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-09-09 17:22:40,521 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-19) Host rack is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-09 17:22:40,552 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-19) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:22:43,570 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to dell-r210ii-16.rhev.lab.eng.brq.redhat.com/10.34.62.208 >2014-09-09 17:22:43,571 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:22:43,571 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:22:43,572 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-2) Exception during connection: java.util.concurrent.ExecutionException: java.net.ConnectException: Connection refused > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:98) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:56) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:57) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:175) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:22:43,575 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-2) Command GetCapabilitiesVDSCommand(HostName = rack, HostId = 1fc56aac-07f1-49db-8972-b5b9567dee77, vds=Host[rack,1fc56aac-07f1-49db-8972-b5b9567dee77]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-09-09 17:22:43,575 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-2) Host rack is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-09 17:22:43,602 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-2) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:22:46,615 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to dell-r210ii-16.rhev.lab.eng.brq.redhat.com/10.34.62.208 >2014-09-09 17:22:46,615 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:22:46,616 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:22:46,616 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-38) Exception during connection: java.util.concurrent.ExecutionException: java.net.ConnectException: Connection refused > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:98) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:56) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:57) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:175) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:22:46,619 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-38) Command GetCapabilitiesVDSCommand(HostName = rack, HostId = 1fc56aac-07f1-49db-8972-b5b9567dee77, vds=Host[rack,1fc56aac-07f1-49db-8972-b5b9567dee77]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-09-09 17:22:46,620 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-38) Host rack is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-09 17:22:46,653 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-38) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:22:49,667 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to dell-r210ii-16.rhev.lab.eng.brq.redhat.com/10.34.62.208 >2014-09-09 17:22:49,668 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:22:49,668 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:22:49,669 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-47) Exception during connection: java.util.concurrent.ExecutionException: java.net.ConnectException: Connection refused > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:98) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:56) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:57) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:175) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:22:49,671 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-47) Command GetCapabilitiesVDSCommand(HostName = rack, HostId = 1fc56aac-07f1-49db-8972-b5b9567dee77, vds=Host[rack,1fc56aac-07f1-49db-8972-b5b9567dee77]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-09-09 17:22:49,671 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-47) Host rack is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-09 17:22:49,697 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-47) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:22:52,712 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to dell-r210ii-16.rhev.lab.eng.brq.redhat.com/10.34.62.208 >2014-09-09 17:22:52,713 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:22:52,713 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:22:52,714 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-50) Exception during connection: java.util.concurrent.ExecutionException: java.net.ConnectException: Connection refused > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:98) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:56) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:57) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:175) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:22:52,717 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-50) Command GetCapabilitiesVDSCommand(HostName = rack, HostId = 1fc56aac-07f1-49db-8972-b5b9567dee77, vds=Host[rack,1fc56aac-07f1-49db-8972-b5b9567dee77]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-09-09 17:22:52,717 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-50) Host rack is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-09 17:22:52,744 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-50) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:22:55,770 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to dell-r210ii-16.rhev.lab.eng.brq.redhat.com/10.34.62.208 >2014-09-09 17:22:55,771 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:22:55,771 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:22:55,772 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-40) Exception during connection: java.util.concurrent.ExecutionException: java.net.ConnectException: Connection refused > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:98) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:56) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:57) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:175) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:22:55,778 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-40) Command GetCapabilitiesVDSCommand(HostName = rack, HostId = 1fc56aac-07f1-49db-8972-b5b9567dee77, vds=Host[rack,1fc56aac-07f1-49db-8972-b5b9567dee77]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-09-09 17:22:55,779 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-40) Host rack is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-09 17:22:55,807 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-40) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:22:58,823 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to dell-r210ii-16.rhev.lab.eng.brq.redhat.com/10.34.62.208 >2014-09-09 17:22:58,823 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:22:58,824 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:22:58,825 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-76) Exception during connection: java.util.concurrent.ExecutionException: java.net.ConnectException: Connection refused > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:98) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:56) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:57) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:175) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:22:58,828 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-76) Command GetCapabilitiesVDSCommand(HostName = rack, HostId = 1fc56aac-07f1-49db-8972-b5b9567dee77, vds=Host[rack,1fc56aac-07f1-49db-8972-b5b9567dee77]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-09-09 17:22:58,828 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-76) Host rack is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-09 17:22:58,855 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-76) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:23:01,869 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to dell-r210ii-16.rhev.lab.eng.brq.redhat.com/10.34.62.208 >2014-09-09 17:23:01,870 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:23:01,871 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:23:01,871 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-77) Exception during connection: java.util.concurrent.ExecutionException: java.net.ConnectException: Connection refused > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:98) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:56) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:57) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:175) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:23:01,873 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-77) [1d4fb057] Command GetCapabilitiesVDSCommand(HostName = rack, HostId = 1fc56aac-07f1-49db-8972-b5b9567dee77, vds=Host[rack,1fc56aac-07f1-49db-8972-b5b9567dee77]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-09-09 17:23:01,874 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-77) [1d4fb057] Host rack is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-09 17:23:01,905 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-77) [1d4fb057] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:23:04,918 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to dell-r210ii-16.rhev.lab.eng.brq.redhat.com/10.34.62.208 >2014-09-09 17:23:04,918 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:23:04,919 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:23:04,919 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-84) Exception during connection: java.util.concurrent.ExecutionException: java.net.ConnectException: Connection refused > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:98) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:56) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:57) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:175) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:23:04,922 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-84) Command GetCapabilitiesVDSCommand(HostName = rack, HostId = 1fc56aac-07f1-49db-8972-b5b9567dee77, vds=Host[rack,1fc56aac-07f1-49db-8972-b5b9567dee77]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-09-09 17:23:04,923 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-84) Host rack is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-09 17:23:04,951 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-84) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:23:07,965 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to dell-r210ii-16.rhev.lab.eng.brq.redhat.com/10.34.62.208 >2014-09-09 17:23:07,966 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:23:07,966 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:23:07,967 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-91) Exception during connection: java.util.concurrent.ExecutionException: java.net.ConnectException: Connection refused > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:98) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:56) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:57) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:175) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:23:07,969 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-91) Command GetCapabilitiesVDSCommand(HostName = rack, HostId = 1fc56aac-07f1-49db-8972-b5b9567dee77, vds=Host[rack,1fc56aac-07f1-49db-8972-b5b9567dee77]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-09-09 17:23:07,969 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-91) Host rack is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-09 17:23:07,996 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-91) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:23:11,012 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to dell-r210ii-16.rhev.lab.eng.brq.redhat.com/10.34.62.208 >2014-09-09 17:23:11,012 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:23:11,013 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:23:11,013 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-97) Exception during connection: java.util.concurrent.ExecutionException: java.net.ConnectException: Connection refused > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:98) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:56) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:57) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:175) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:23:11,015 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-97) Command GetCapabilitiesVDSCommand(HostName = rack, HostId = 1fc56aac-07f1-49db-8972-b5b9567dee77, vds=Host[rack,1fc56aac-07f1-49db-8972-b5b9567dee77]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-09-09 17:23:11,015 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-97) Host rack is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-09 17:23:11,042 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-97) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:23:14,058 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to dell-r210ii-16.rhev.lab.eng.brq.redhat.com/10.34.62.208 >2014-09-09 17:23:14,059 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:23:14,059 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:23:14,060 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-12) Exception during connection: java.util.concurrent.ExecutionException: java.net.ConnectException: Connection refused > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:98) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:56) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:57) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:175) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:23:14,064 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-12) Command GetCapabilitiesVDSCommand(HostName = rack, HostId = 1fc56aac-07f1-49db-8972-b5b9567dee77, vds=Host[rack,1fc56aac-07f1-49db-8972-b5b9567dee77]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-09-09 17:23:14,064 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-12) Host rack is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-09 17:23:14,092 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-12) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:23:17,105 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to dell-r210ii-16.rhev.lab.eng.brq.redhat.com/10.34.62.208 >2014-09-09 17:23:17,105 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:23:17,106 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:23:17,106 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-11) Exception during connection: java.util.concurrent.ExecutionException: java.net.ConnectException: Connection refused > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:98) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:56) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:57) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:175) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:23:17,109 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-11) Command GetCapabilitiesVDSCommand(HostName = rack, HostId = 1fc56aac-07f1-49db-8972-b5b9567dee77, vds=Host[rack,1fc56aac-07f1-49db-8972-b5b9567dee77]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-09-09 17:23:17,109 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-11) Host rack is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-09 17:23:17,137 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-11) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:23:20,150 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to dell-r210ii-16.rhev.lab.eng.brq.redhat.com/10.34.62.208 >2014-09-09 17:23:20,151 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:23:20,151 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:23:20,152 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-24) Exception during connection: java.util.concurrent.ExecutionException: java.net.ConnectException: Connection refused > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:98) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:56) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:57) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:175) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:23:20,155 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-24) Command GetCapabilitiesVDSCommand(HostName = rack, HostId = 1fc56aac-07f1-49db-8972-b5b9567dee77, vds=Host[rack,1fc56aac-07f1-49db-8972-b5b9567dee77]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-09-09 17:23:20,162 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-24) Host rack is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-09 17:23:20,195 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-24) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:23:23,211 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to dell-r210ii-16.rhev.lab.eng.brq.redhat.com/10.34.62.208 >2014-09-09 17:23:23,212 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:23:23,212 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:23:23,213 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-19) Exception during connection: java.util.concurrent.ExecutionException: java.net.ConnectException: Connection refused > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:98) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:56) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:57) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:175) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:23:23,215 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-19) Command GetCapabilitiesVDSCommand(HostName = rack, HostId = 1fc56aac-07f1-49db-8972-b5b9567dee77, vds=Host[rack,1fc56aac-07f1-49db-8972-b5b9567dee77]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-09-09 17:23:23,215 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-19) Host rack is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-09 17:23:23,244 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-19) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:23:26,257 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to dell-r210ii-16.rhev.lab.eng.brq.redhat.com/10.34.62.208 >2014-09-09 17:23:26,258 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:23:26,258 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:23:26,259 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-44) Exception during connection: java.util.concurrent.ExecutionException: java.net.ConnectException: Connection refused > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:98) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:56) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:57) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:175) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:23:26,261 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-44) Command GetCapabilitiesVDSCommand(HostName = rack, HostId = 1fc56aac-07f1-49db-8972-b5b9567dee77, vds=Host[rack,1fc56aac-07f1-49db-8972-b5b9567dee77]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-09-09 17:23:26,261 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-44) Host rack is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-09 17:23:26,314 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-44) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:23:29,328 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to dell-r210ii-16.rhev.lab.eng.brq.redhat.com/10.34.62.208 >2014-09-09 17:23:29,329 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:23:29,330 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:23:29,330 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-41) Exception during connection: java.util.concurrent.ExecutionException: java.net.ConnectException: Connection refused > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:98) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:56) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:57) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:175) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:23:29,333 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-41) Command GetCapabilitiesVDSCommand(HostName = rack, HostId = 1fc56aac-07f1-49db-8972-b5b9567dee77, vds=Host[rack,1fc56aac-07f1-49db-8972-b5b9567dee77]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-09-09 17:23:29,333 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-41) Host rack is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-09 17:23:29,363 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-41) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:23:32,378 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to dell-r210ii-16.rhev.lab.eng.brq.redhat.com/10.34.62.208 >2014-09-09 17:23:32,379 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:23:32,379 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:23:32,379 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-53) Exception during connection: java.util.concurrent.ExecutionException: java.net.ConnectException: Connection refused > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:98) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:56) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:57) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:175) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:23:32,381 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-53) [5733dd3e] Command GetCapabilitiesVDSCommand(HostName = rack, HostId = 1fc56aac-07f1-49db-8972-b5b9567dee77, vds=Host[rack,1fc56aac-07f1-49db-8972-b5b9567dee77]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-09-09 17:23:32,382 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-53) [5733dd3e] Host rack is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-09 17:23:32,411 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-53) [5733dd3e] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:23:35,425 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to dell-r210ii-16.rhev.lab.eng.brq.redhat.com/10.34.62.208 >2014-09-09 17:23:35,426 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:23:35,426 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:23:35,427 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-62) Exception during connection: java.util.concurrent.ExecutionException: java.net.ConnectException: Connection refused > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:98) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:56) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:57) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:175) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:23:35,440 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-62) Command GetCapabilitiesVDSCommand(HostName = rack, HostId = 1fc56aac-07f1-49db-8972-b5b9567dee77, vds=Host[rack,1fc56aac-07f1-49db-8972-b5b9567dee77]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-09-09 17:23:35,444 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-62) Host rack is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-09 17:23:35,470 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-62) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:23:38,486 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to dell-r210ii-16.rhev.lab.eng.brq.redhat.com/10.34.62.208 >2014-09-09 17:23:38,486 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:23:38,487 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:23:38,487 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-63) Exception during connection: java.util.concurrent.ExecutionException: java.net.ConnectException: Connection refused > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:98) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:56) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:57) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:175) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:23:38,489 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-63) Command GetCapabilitiesVDSCommand(HostName = rack, HostId = 1fc56aac-07f1-49db-8972-b5b9567dee77, vds=Host[rack,1fc56aac-07f1-49db-8972-b5b9567dee77]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-09-09 17:23:38,490 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-63) Host rack is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-09 17:23:38,555 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-63) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:23:41,571 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to dell-r210ii-16.rhev.lab.eng.brq.redhat.com/10.34.62.208 >2014-09-09 17:23:41,571 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:23:41,572 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:23:41,572 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-66) Exception during connection: java.util.concurrent.ExecutionException: java.net.ConnectException: Connection refused > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:98) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:56) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:57) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:175) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:23:41,574 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-66) [266d3fbf] Command GetCapabilitiesVDSCommand(HostName = rack, HostId = 1fc56aac-07f1-49db-8972-b5b9567dee77, vds=Host[rack,1fc56aac-07f1-49db-8972-b5b9567dee77]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-09-09 17:23:41,575 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-66) [266d3fbf] Host rack is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-09 17:23:41,601 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-66) [266d3fbf] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:23:44,620 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to dell-r210ii-16.rhev.lab.eng.brq.redhat.com/10.34.62.208 >2014-09-09 17:23:44,621 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:23:44,621 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:23:44,624 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-71) Exception during connection: java.util.concurrent.ExecutionException: java.net.ConnectException: Connection refused > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:98) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:56) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:57) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:175) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:23:44,627 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-71) Command GetCapabilitiesVDSCommand(HostName = rack, HostId = 1fc56aac-07f1-49db-8972-b5b9567dee77, vds=Host[rack,1fc56aac-07f1-49db-8972-b5b9567dee77]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-09-09 17:23:44,627 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-71) Host rack is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-09 17:23:44,655 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-71) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:23:47,670 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to dell-r210ii-16.rhev.lab.eng.brq.redhat.com/10.34.62.208 >2014-09-09 17:23:47,671 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:23:47,671 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:23:47,671 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-77) Exception during connection: java.util.concurrent.ExecutionException: java.net.ConnectException: Connection refused > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:98) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:56) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:57) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:175) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:23:47,674 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-77) [1d4fb057] Command GetCapabilitiesVDSCommand(HostName = rack, HostId = 1fc56aac-07f1-49db-8972-b5b9567dee77, vds=Host[rack,1fc56aac-07f1-49db-8972-b5b9567dee77]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-09-09 17:23:47,674 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-77) [1d4fb057] Host rack is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-09 17:23:47,738 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-77) [1d4fb057] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:23:50,751 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to dell-r210ii-16.rhev.lab.eng.brq.redhat.com/10.34.62.208 >2014-09-09 17:23:50,752 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:23:50,752 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:23:50,753 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-90) Exception during connection: java.util.concurrent.ExecutionException: java.net.ConnectException: Connection refused > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:98) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:56) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:57) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:175) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:23:50,755 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-90) [18a5c631] Command GetCapabilitiesVDSCommand(HostName = rack, HostId = 1fc56aac-07f1-49db-8972-b5b9567dee77, vds=Host[rack,1fc56aac-07f1-49db-8972-b5b9567dee77]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-09-09 17:23:50,755 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-90) [18a5c631] Host rack is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-09 17:23:50,781 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-90) [18a5c631] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:23:53,809 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to dell-r210ii-16.rhev.lab.eng.brq.redhat.com/10.34.62.208 >2014-09-09 17:23:53,810 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:23:53,810 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:23:53,811 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-16) Exception during connection: java.util.concurrent.ExecutionException: java.net.ConnectException: Connection refused > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:98) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:56) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:57) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:175) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:23:53,814 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-16) Command GetCapabilitiesVDSCommand(HostName = rack, HostId = 1fc56aac-07f1-49db-8972-b5b9567dee77, vds=Host[rack,1fc56aac-07f1-49db-8972-b5b9567dee77]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-09-09 17:23:53,814 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-16) Host rack is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-09 17:23:53,843 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-16) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:23:56,865 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to dell-r210ii-16.rhev.lab.eng.brq.redhat.com/10.34.62.208 >2014-09-09 17:23:56,866 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:23:56,867 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:23:56,867 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-7) Exception during connection: java.util.concurrent.ExecutionException: java.net.ConnectException: Connection refused > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:98) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:56) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:57) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:175) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:23:56,870 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-7) Command GetCapabilitiesVDSCommand(HostName = rack, HostId = 1fc56aac-07f1-49db-8972-b5b9567dee77, vds=Host[rack,1fc56aac-07f1-49db-8972-b5b9567dee77]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-09-09 17:23:56,870 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-7) Host rack is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-09 17:23:56,899 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-7) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:23:59,917 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to dell-r210ii-16.rhev.lab.eng.brq.redhat.com/10.34.62.208 >2014-09-09 17:23:59,918 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:23:59,918 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:23:59,919 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-14) Exception during connection: java.util.concurrent.ExecutionException: java.net.ConnectException: Connection refused > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:98) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:56) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:57) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:175) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:23:59,922 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-14) Command GetCapabilitiesVDSCommand(HostName = rack, HostId = 1fc56aac-07f1-49db-8972-b5b9567dee77, vds=Host[rack,1fc56aac-07f1-49db-8972-b5b9567dee77]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-09-09 17:23:59,922 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-14) Host rack is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-09 17:23:59,961 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-14) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:24:02,979 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to dell-r210ii-16.rhev.lab.eng.brq.redhat.com/10.34.62.208 >2014-09-09 17:24:02,980 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:24:02,980 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:24:02,980 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-30) Exception during connection: java.util.concurrent.ExecutionException: java.net.ConnectException: Connection refused > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:98) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:56) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:57) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:175) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:24:02,984 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-30) [5b453678] Command GetCapabilitiesVDSCommand(HostName = rack, HostId = 1fc56aac-07f1-49db-8972-b5b9567dee77, vds=Host[rack,1fc56aac-07f1-49db-8972-b5b9567dee77]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-09-09 17:24:02,984 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-30) [5b453678] Host rack is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-09 17:24:03,011 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-30) [5b453678] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:24:06,026 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to dell-r210ii-16.rhev.lab.eng.brq.redhat.com/10.34.62.208 >2014-09-09 17:24:06,027 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:24:06,028 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:24:06,028 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-28) Exception during connection: java.util.concurrent.ExecutionException: java.net.ConnectException: Connection refused > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:98) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:56) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:57) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:175) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:24:06,031 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-28) Command GetCapabilitiesVDSCommand(HostName = rack, HostId = 1fc56aac-07f1-49db-8972-b5b9567dee77, vds=Host[rack,1fc56aac-07f1-49db-8972-b5b9567dee77]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-09-09 17:24:06,031 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-28) Host rack is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-09 17:24:06,071 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-28) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:24:09,087 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to dell-r210ii-16.rhev.lab.eng.brq.redhat.com/10.34.62.208 >2014-09-09 17:24:09,088 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:24:09,088 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:24:09,089 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-19) Exception during connection: java.util.concurrent.ExecutionException: java.net.ConnectException: Connection refused > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:98) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:56) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:57) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:175) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:24:09,091 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-19) Command GetCapabilitiesVDSCommand(HostName = rack, HostId = 1fc56aac-07f1-49db-8972-b5b9567dee77, vds=Host[rack,1fc56aac-07f1-49db-8972-b5b9567dee77]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-09-09 17:24:09,091 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-19) Host rack is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-09 17:24:09,117 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-19) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:24:12,131 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to dell-r210ii-16.rhev.lab.eng.brq.redhat.com/10.34.62.208 >2014-09-09 17:24:12,131 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:24:12,132 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:24:12,133 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-39) Exception during connection: java.util.concurrent.ExecutionException: java.net.ConnectException: Connection refused > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:98) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:56) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:57) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:175) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:24:12,136 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-39) Command GetCapabilitiesVDSCommand(HostName = rack, HostId = 1fc56aac-07f1-49db-8972-b5b9567dee77, vds=Host[rack,1fc56aac-07f1-49db-8972-b5b9567dee77]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-09-09 17:24:12,136 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-39) Host rack is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-09 17:24:12,167 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-39) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:24:15,182 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to dell-r210ii-16.rhev.lab.eng.brq.redhat.com/10.34.62.208 >2014-09-09 17:24:15,183 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:24:15,183 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:24:15,183 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-49) Exception during connection: java.util.concurrent.ExecutionException: java.net.ConnectException: Connection refused > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:98) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:56) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:57) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:175) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:24:15,187 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-49) [7e37bc27] Command GetCapabilitiesVDSCommand(HostName = rack, HostId = 1fc56aac-07f1-49db-8972-b5b9567dee77, vds=Host[rack,1fc56aac-07f1-49db-8972-b5b9567dee77]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-09-09 17:24:15,187 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-49) [7e37bc27] Host rack is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-09 17:24:15,217 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-49) [7e37bc27] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:24:18,236 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to dell-r210ii-16.rhev.lab.eng.brq.redhat.com/10.34.62.208 >2014-09-09 17:24:18,237 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:24:18,237 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:24:18,238 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-51) Exception during connection: java.util.concurrent.ExecutionException: java.net.ConnectException: Connection refused > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:98) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:56) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:57) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:175) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:24:18,246 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-51) [5c5dd985] Command GetCapabilitiesVDSCommand(HostName = rack, HostId = 1fc56aac-07f1-49db-8972-b5b9567dee77, vds=Host[rack,1fc56aac-07f1-49db-8972-b5b9567dee77]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-09-09 17:24:18,246 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-51) [5c5dd985] Host rack is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-09 17:24:18,278 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-51) [5c5dd985] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:24:21,295 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to dell-r210ii-16.rhev.lab.eng.brq.redhat.com/10.34.62.208 >2014-09-09 17:24:21,296 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:24:21,296 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:24:21,297 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-68) Exception during connection: java.util.concurrent.ExecutionException: java.net.ConnectException: Connection refused > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:98) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:56) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:57) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:175) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:24:21,299 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-68) Command GetCapabilitiesVDSCommand(HostName = rack, HostId = 1fc56aac-07f1-49db-8972-b5b9567dee77, vds=Host[rack,1fc56aac-07f1-49db-8972-b5b9567dee77]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-09-09 17:24:21,300 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-68) Host rack is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-09 17:24:21,335 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-68) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:24:24,353 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to dell-r210ii-16.rhev.lab.eng.brq.redhat.com/10.34.62.208 >2014-09-09 17:24:24,354 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:24:24,354 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:24:24,354 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-65) Exception during connection: java.util.concurrent.ExecutionException: java.net.ConnectException: Connection refused > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:98) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:56) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:57) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:175) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:24:24,357 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-65) Command GetCapabilitiesVDSCommand(HostName = rack, HostId = 1fc56aac-07f1-49db-8972-b5b9567dee77, vds=Host[rack,1fc56aac-07f1-49db-8972-b5b9567dee77]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-09-09 17:24:24,357 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-65) Host rack is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-09 17:24:24,397 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-65) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:24:27,418 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to dell-r210ii-16.rhev.lab.eng.brq.redhat.com/10.34.62.208 >2014-09-09 17:24:27,418 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:24:27,419 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:24:27,419 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-73) Exception during connection: java.util.concurrent.ExecutionException: java.net.ConnectException: Connection refused > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:98) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:56) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:57) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:175) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:24:27,422 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-73) Command GetCapabilitiesVDSCommand(HostName = rack, HostId = 1fc56aac-07f1-49db-8972-b5b9567dee77, vds=Host[rack,1fc56aac-07f1-49db-8972-b5b9567dee77]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-09-09 17:24:27,422 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-73) Host rack is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-09 17:24:27,455 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-73) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:24:30,467 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to dell-r210ii-16.rhev.lab.eng.brq.redhat.com/10.34.62.208 >2014-09-09 17:24:30,467 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:24:30,468 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:24:30,468 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-75) Exception during connection: java.util.concurrent.ExecutionException: java.net.ConnectException: Connection refused > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:98) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:56) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:57) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:175) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:24:30,472 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-75) Command GetCapabilitiesVDSCommand(HostName = rack, HostId = 1fc56aac-07f1-49db-8972-b5b9567dee77, vds=Host[rack,1fc56aac-07f1-49db-8972-b5b9567dee77]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-09-09 17:24:30,472 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-75) Host rack is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-09 17:24:30,494 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-75) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:24:33,505 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to dell-r210ii-16.rhev.lab.eng.brq.redhat.com/10.34.62.208 >2014-09-09 17:24:33,505 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:24:33,506 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:24:33,506 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-85) Exception during connection: java.util.concurrent.ExecutionException: java.net.ConnectException: Connection refused > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:98) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:56) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:57) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:175) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:24:33,509 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-85) Command GetCapabilitiesVDSCommand(HostName = rack, HostId = 1fc56aac-07f1-49db-8972-b5b9567dee77, vds=Host[rack,1fc56aac-07f1-49db-8972-b5b9567dee77]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-09-09 17:24:33,510 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-85) Host rack is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-09 17:24:33,547 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-85) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:24:36,566 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to dell-r210ii-16.rhev.lab.eng.brq.redhat.com/10.34.62.208 >2014-09-09 17:24:36,570 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:24:36,571 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:24:36,571 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-1) Exception during connection: java.util.concurrent.ExecutionException: java.net.ConnectException: Connection refused > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:98) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:56) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:57) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:175) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:24:36,574 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-1) Command GetCapabilitiesVDSCommand(HostName = rack, HostId = 1fc56aac-07f1-49db-8972-b5b9567dee77, vds=Host[rack,1fc56aac-07f1-49db-8972-b5b9567dee77]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-09-09 17:24:36,576 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-1) Host rack is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-09 17:24:36,607 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-1) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:24:39,625 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to dell-r210ii-16.rhev.lab.eng.brq.redhat.com/10.34.62.208 >2014-09-09 17:24:39,626 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:24:39,626 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:24:39,627 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-96) Exception during connection: java.util.concurrent.ExecutionException: java.net.ConnectException: Connection refused > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:98) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:56) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:57) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:175) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:24:39,629 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-96) Command GetCapabilitiesVDSCommand(HostName = rack, HostId = 1fc56aac-07f1-49db-8972-b5b9567dee77, vds=Host[rack,1fc56aac-07f1-49db-8972-b5b9567dee77]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-09-09 17:24:39,630 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-96) Host rack is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-09 17:24:39,667 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-96) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:24:42,686 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to dell-r210ii-16.rhev.lab.eng.brq.redhat.com/10.34.62.208 >2014-09-09 17:24:42,687 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:24:42,687 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:24:42,688 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-3) Exception during connection: java.util.concurrent.ExecutionException: java.net.ConnectException: Connection refused > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:98) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:56) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:57) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:175) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:24:42,691 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-3) Command GetCapabilitiesVDSCommand(HostName = rack, HostId = 1fc56aac-07f1-49db-8972-b5b9567dee77, vds=Host[rack,1fc56aac-07f1-49db-8972-b5b9567dee77]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-09-09 17:24:42,691 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-3) Host rack is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-09 17:24:42,708 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-3) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host rack is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-09 17:24:42,734 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-3) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:24:45,748 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to dell-r210ii-16.rhev.lab.eng.brq.redhat.com/10.34.62.208 >2014-09-09 17:24:45,749 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:24:45,750 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:24:45,750 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-14) Exception during connection: java.util.concurrent.ExecutionException: java.net.ConnectException: Connection refused > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:98) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:56) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:57) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:175) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:24:45,753 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-14) Command GetCapabilitiesVDSCommand(HostName = rack, HostId = 1fc56aac-07f1-49db-8972-b5b9567dee77, vds=Host[rack,1fc56aac-07f1-49db-8972-b5b9567dee77]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-09-09 17:24:45,753 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-14) Host rack is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-09 17:24:45,783 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-14) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:24:48,798 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to dell-r210ii-16.rhev.lab.eng.brq.redhat.com/10.34.62.208 >2014-09-09 17:24:48,798 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:24:48,799 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:24:48,799 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-20) Exception during connection: java.util.concurrent.ExecutionException: java.net.ConnectException: Connection refused > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:98) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:56) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:57) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:175) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:24:48,802 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-20) Command GetCapabilitiesVDSCommand(HostName = rack, HostId = 1fc56aac-07f1-49db-8972-b5b9567dee77, vds=Host[rack,1fc56aac-07f1-49db-8972-b5b9567dee77]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-09-09 17:24:48,802 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-20) Host rack is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-09 17:24:48,835 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-20) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:24:51,849 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to dell-r210ii-16.rhev.lab.eng.brq.redhat.com/10.34.62.208 >2014-09-09 17:24:51,850 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:24:51,850 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:24:51,851 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-25) Exception during connection: java.util.concurrent.ExecutionException: java.net.ConnectException: Connection refused > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:98) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:56) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:57) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:175) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:24:51,853 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-25) [27472998] Command GetCapabilitiesVDSCommand(HostName = rack, HostId = 1fc56aac-07f1-49db-8972-b5b9567dee77, vds=Host[rack,1fc56aac-07f1-49db-8972-b5b9567dee77]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-09-09 17:24:51,854 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-25) [27472998] Host rack is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-09 17:24:51,883 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-25) [27472998] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:24:54,898 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to dell-r210ii-16.rhev.lab.eng.brq.redhat.com/10.34.62.208 >2014-09-09 17:24:54,899 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:24:54,900 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:24:54,900 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-39) Exception during connection: java.util.concurrent.ExecutionException: java.net.ConnectException: Connection refused > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:98) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:56) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:57) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:175) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:24:54,903 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-39) Command GetCapabilitiesVDSCommand(HostName = rack, HostId = 1fc56aac-07f1-49db-8972-b5b9567dee77, vds=Host[rack,1fc56aac-07f1-49db-8972-b5b9567dee77]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-09-09 17:24:54,903 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-39) Host rack is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-09 17:24:54,952 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-39) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:24:57,967 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to dell-r210ii-16.rhev.lab.eng.brq.redhat.com/10.34.62.208 >2014-09-09 17:24:57,968 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:24:57,968 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:24:57,969 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-38) Exception during connection: java.util.concurrent.ExecutionException: java.net.ConnectException: Connection refused > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:98) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:56) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:57) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:175) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:24:57,971 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-38) Command GetCapabilitiesVDSCommand(HostName = rack, HostId = 1fc56aac-07f1-49db-8972-b5b9567dee77, vds=Host[rack,1fc56aac-07f1-49db-8972-b5b9567dee77]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-09-09 17:24:57,971 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-38) Host rack is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-09 17:24:57,997 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-38) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:25:01,012 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to dell-r210ii-16.rhev.lab.eng.brq.redhat.com/10.34.62.208 >2014-09-09 17:25:01,013 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:25:01,013 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:25:01,014 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-52) Exception during connection: java.util.concurrent.ExecutionException: java.net.ConnectException: Connection refused > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:98) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:56) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:57) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:175) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:25:01,017 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-52) Command GetCapabilitiesVDSCommand(HostName = rack, HostId = 1fc56aac-07f1-49db-8972-b5b9567dee77, vds=Host[rack,1fc56aac-07f1-49db-8972-b5b9567dee77]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-09-09 17:25:01,017 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-52) Host rack is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-09 17:25:01,050 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-52) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:25:04,062 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to dell-r210ii-16.rhev.lab.eng.brq.redhat.com/10.34.62.208 >2014-09-09 17:25:04,063 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:25:04,063 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:25:04,063 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-62) Exception during connection: java.util.concurrent.ExecutionException: java.net.ConnectException: Connection refused > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:98) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:56) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:57) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:175) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:25:04,065 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-62) Command GetCapabilitiesVDSCommand(HostName = rack, HostId = 1fc56aac-07f1-49db-8972-b5b9567dee77, vds=Host[rack,1fc56aac-07f1-49db-8972-b5b9567dee77]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-09-09 17:25:04,065 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-62) Host rack is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-09 17:25:04,086 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-62) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:25:07,097 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to dell-r210ii-16.rhev.lab.eng.brq.redhat.com/10.34.62.208 >2014-09-09 17:25:07,098 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:25:07,098 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:25:07,098 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-67) Exception during connection: java.util.concurrent.ExecutionException: java.net.ConnectException: Connection refused > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:98) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:56) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:57) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:175) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:25:07,101 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-67) Command GetCapabilitiesVDSCommand(HostName = rack, HostId = 1fc56aac-07f1-49db-8972-b5b9567dee77, vds=Host[rack,1fc56aac-07f1-49db-8972-b5b9567dee77]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-09-09 17:25:07,101 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-67) Host rack is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-09 17:25:07,126 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-67) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:25:10,135 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to dell-r210ii-16.rhev.lab.eng.brq.redhat.com/10.34.62.208 >2014-09-09 17:25:10,135 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:25:10,135 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:25:10,138 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-40) Exception during connection: java.util.concurrent.ExecutionException: java.net.ConnectException: Connection refused > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:98) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:56) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:57) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:175) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:25:10,150 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-40) Command GetCapabilitiesVDSCommand(HostName = rack, HostId = 1fc56aac-07f1-49db-8972-b5b9567dee77, vds=Host[rack,1fc56aac-07f1-49db-8972-b5b9567dee77]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-09-09 17:25:10,150 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-40) Host rack is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-09 17:25:10,172 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-40) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:25:13,190 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to dell-r210ii-16.rhev.lab.eng.brq.redhat.com/10.34.62.208 >2014-09-09 17:25:13,191 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:25:13,192 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:25:13,192 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-80) Exception during connection: java.util.concurrent.ExecutionException: java.net.ConnectException: Connection refused > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:98) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:56) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:57) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:175) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:25:13,195 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-80) Command GetCapabilitiesVDSCommand(HostName = rack, HostId = 1fc56aac-07f1-49db-8972-b5b9567dee77, vds=Host[rack,1fc56aac-07f1-49db-8972-b5b9567dee77]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-09-09 17:25:13,195 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-80) Host rack is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-09 17:25:13,225 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-80) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:25:16,246 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to dell-r210ii-16.rhev.lab.eng.brq.redhat.com/10.34.62.208 >2014-09-09 17:25:16,247 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:25:16,248 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:25:16,248 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-83) Exception during connection: java.util.concurrent.ExecutionException: java.net.ConnectException: Connection refused > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:98) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:56) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:57) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:175) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:25:16,251 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-83) [b670564] Command GetCapabilitiesVDSCommand(HostName = rack, HostId = 1fc56aac-07f1-49db-8972-b5b9567dee77, vds=Host[rack,1fc56aac-07f1-49db-8972-b5b9567dee77]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-09-09 17:25:16,251 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-83) [b670564] Host rack is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-09 17:25:16,282 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-83) [b670564] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:25:19,297 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to dell-r210ii-16.rhev.lab.eng.brq.redhat.com/10.34.62.208 >2014-09-09 17:25:19,298 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:25:19,299 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:25:19,299 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-90) Exception during connection: java.util.concurrent.ExecutionException: java.net.ConnectException: Connection refused > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:98) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:56) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:57) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:175) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:25:19,302 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-90) [18a5c631] Command GetCapabilitiesVDSCommand(HostName = rack, HostId = 1fc56aac-07f1-49db-8972-b5b9567dee77, vds=Host[rack,1fc56aac-07f1-49db-8972-b5b9567dee77]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-09-09 17:25:19,302 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-90) [18a5c631] Host rack is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-09 17:25:19,334 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-90) [18a5c631] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:25:22,355 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to dell-r210ii-16.rhev.lab.eng.brq.redhat.com/10.34.62.208 >2014-09-09 17:25:22,355 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:25:22,356 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:25:22,356 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-86) Exception during connection: java.util.concurrent.ExecutionException: java.net.ConnectException: Connection refused > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:98) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:56) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:57) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:175) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:25:22,360 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-86) Command GetCapabilitiesVDSCommand(HostName = rack, HostId = 1fc56aac-07f1-49db-8972-b5b9567dee77, vds=Host[rack,1fc56aac-07f1-49db-8972-b5b9567dee77]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-09-09 17:25:22,360 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-86) Host rack is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-09 17:25:22,399 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-86) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:25:25,415 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to dell-r210ii-16.rhev.lab.eng.brq.redhat.com/10.34.62.208 >2014-09-09 17:25:25,415 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:25:25,416 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:25:25,416 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-3) Exception during connection: java.util.concurrent.ExecutionException: java.net.ConnectException: Connection refused > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:98) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:56) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:57) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:175) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:25:25,419 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-3) Command GetCapabilitiesVDSCommand(HostName = rack, HostId = 1fc56aac-07f1-49db-8972-b5b9567dee77, vds=Host[rack,1fc56aac-07f1-49db-8972-b5b9567dee77]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-09-09 17:25:25,420 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-3) Host rack is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-09 17:25:25,464 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-3) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:25:28,480 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to dell-r210ii-16.rhev.lab.eng.brq.redhat.com/10.34.62.208 >2014-09-09 17:25:28,480 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:25:28,481 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:25:28,481 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-4) Exception during connection: java.util.concurrent.ExecutionException: java.net.ConnectException: Connection refused > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:98) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:56) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:57) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:175) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:25:28,485 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-4) Command GetCapabilitiesVDSCommand(HostName = rack, HostId = 1fc56aac-07f1-49db-8972-b5b9567dee77, vds=Host[rack,1fc56aac-07f1-49db-8972-b5b9567dee77]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-09-09 17:25:28,486 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-4) Host rack is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-09 17:25:28,517 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-4) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:25:31,533 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to dell-r210ii-16.rhev.lab.eng.brq.redhat.com/10.34.62.208 >2014-09-09 17:25:31,534 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:25:31,535 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:25:31,535 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-20) Exception during connection: java.util.concurrent.ExecutionException: java.net.ConnectException: Connection refused > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:98) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:56) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:57) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:175) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:25:31,544 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-20) Command GetCapabilitiesVDSCommand(HostName = rack, HostId = 1fc56aac-07f1-49db-8972-b5b9567dee77, vds=Host[rack,1fc56aac-07f1-49db-8972-b5b9567dee77]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-09-09 17:25:31,544 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-20) Host rack is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-09 17:25:31,585 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-20) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:25:34,602 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to dell-r210ii-16.rhev.lab.eng.brq.redhat.com/10.34.62.208 >2014-09-09 17:25:34,603 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:25:34,603 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:25:34,604 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-9) Exception during connection: java.util.concurrent.ExecutionException: java.net.ConnectException: Connection refused > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:98) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:56) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:57) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:175) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:25:34,607 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-9) Command GetCapabilitiesVDSCommand(HostName = rack, HostId = 1fc56aac-07f1-49db-8972-b5b9567dee77, vds=Host[rack,1fc56aac-07f1-49db-8972-b5b9567dee77]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-09-09 17:25:34,607 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-9) Host rack is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-09 17:25:34,640 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-9) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:25:37,654 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to dell-r210ii-16.rhev.lab.eng.brq.redhat.com/10.34.62.208 >2014-09-09 17:25:37,655 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:25:37,655 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:25:37,656 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-22) Exception during connection: java.util.concurrent.ExecutionException: java.net.ConnectException: Connection refused > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:98) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:56) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:57) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:175) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:25:37,658 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-22) Command GetCapabilitiesVDSCommand(HostName = rack, HostId = 1fc56aac-07f1-49db-8972-b5b9567dee77, vds=Host[rack,1fc56aac-07f1-49db-8972-b5b9567dee77]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-09-09 17:25:37,658 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-22) Host rack is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-09 17:25:37,691 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-22) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:25:40,710 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to dell-r210ii-16.rhev.lab.eng.brq.redhat.com/10.34.62.208 >2014-09-09 17:25:40,711 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:25:40,711 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:25:40,712 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-44) Exception during connection: java.util.concurrent.ExecutionException: java.net.ConnectException: Connection refused > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:98) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:56) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:57) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:175) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:25:40,714 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-44) Command GetCapabilitiesVDSCommand(HostName = rack, HostId = 1fc56aac-07f1-49db-8972-b5b9567dee77, vds=Host[rack,1fc56aac-07f1-49db-8972-b5b9567dee77]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-09-09 17:25:40,715 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-44) Host rack is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-09 17:25:40,744 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-44) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:25:43,758 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to dell-r210ii-16.rhev.lab.eng.brq.redhat.com/10.34.62.208 >2014-09-09 17:25:43,759 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:25:43,759 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:25:43,760 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-42) Exception during connection: java.util.concurrent.ExecutionException: java.net.ConnectException: Connection refused > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:98) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:56) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:57) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:175) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:25:43,763 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-42) Command GetCapabilitiesVDSCommand(HostName = rack, HostId = 1fc56aac-07f1-49db-8972-b5b9567dee77, vds=Host[rack,1fc56aac-07f1-49db-8972-b5b9567dee77]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-09-09 17:25:43,768 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-42) Host rack is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-09 17:25:43,808 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-42) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:25:46,825 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to dell-r210ii-16.rhev.lab.eng.brq.redhat.com/10.34.62.208 >2014-09-09 17:25:46,825 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:25:46,826 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:25:46,827 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-52) Exception during connection: java.util.concurrent.ExecutionException: java.net.ConnectException: Connection refused > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:98) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:56) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:57) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:175) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:25:46,829 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-52) Command GetCapabilitiesVDSCommand(HostName = rack, HostId = 1fc56aac-07f1-49db-8972-b5b9567dee77, vds=Host[rack,1fc56aac-07f1-49db-8972-b5b9567dee77]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-09-09 17:25:46,830 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-52) Host rack is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-09 17:25:46,858 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-52) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:25:49,873 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to dell-r210ii-16.rhev.lab.eng.brq.redhat.com/10.34.62.208 >2014-09-09 17:25:49,874 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:25:49,874 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:25:49,875 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-62) Exception during connection: java.util.concurrent.ExecutionException: java.net.ConnectException: Connection refused > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:98) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:56) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:57) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:175) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:25:49,877 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-62) Command GetCapabilitiesVDSCommand(HostName = rack, HostId = 1fc56aac-07f1-49db-8972-b5b9567dee77, vds=Host[rack,1fc56aac-07f1-49db-8972-b5b9567dee77]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-09-09 17:25:49,878 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-62) Host rack is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-09 17:25:49,906 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-62) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:25:52,923 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to dell-r210ii-16.rhev.lab.eng.brq.redhat.com/10.34.62.208 >2014-09-09 17:25:52,924 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:25:52,925 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:25:52,925 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-72) Exception during connection: java.util.concurrent.ExecutionException: java.net.ConnectException: Connection refused > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:98) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:56) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:57) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:175) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:25:52,928 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-72) [3822e820] Command GetCapabilitiesVDSCommand(HostName = rack, HostId = 1fc56aac-07f1-49db-8972-b5b9567dee77, vds=Host[rack,1fc56aac-07f1-49db-8972-b5b9567dee77]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-09-09 17:25:52,928 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-72) [3822e820] Host rack is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-09 17:25:52,975 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-72) [3822e820] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:25:55,988 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to dell-r210ii-16.rhev.lab.eng.brq.redhat.com/10.34.62.208 >2014-09-09 17:25:55,989 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:25:55,990 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:25:55,990 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-76) Exception during connection: java.util.concurrent.ExecutionException: java.net.ConnectException: Connection refused > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:98) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:56) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:57) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:175) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:25:55,992 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-76) Command GetCapabilitiesVDSCommand(HostName = rack, HostId = 1fc56aac-07f1-49db-8972-b5b9567dee77, vds=Host[rack,1fc56aac-07f1-49db-8972-b5b9567dee77]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-09-09 17:25:55,992 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-76) Host rack is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-09 17:25:56,024 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-76) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:25:59,042 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to dell-r210ii-16.rhev.lab.eng.brq.redhat.com/10.34.62.208 >2014-09-09 17:25:59,046 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:25:59,046 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:25:59,047 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-81) Exception during connection: java.util.concurrent.ExecutionException: java.net.ConnectException: Connection refused > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:98) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:56) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:57) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:175) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:25:59,050 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-81) [24b7b64b] Command GetCapabilitiesVDSCommand(HostName = rack, HostId = 1fc56aac-07f1-49db-8972-b5b9567dee77, vds=Host[rack,1fc56aac-07f1-49db-8972-b5b9567dee77]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-09-09 17:25:59,050 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-81) [24b7b64b] Host rack is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-09 17:25:59,083 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-81) [24b7b64b] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:26:02,098 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to dell-r210ii-16.rhev.lab.eng.brq.redhat.com/10.34.62.208 >2014-09-09 17:26:02,098 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:26:02,099 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:26:02,099 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-82) Exception during connection: java.util.concurrent.ExecutionException: java.net.ConnectException: Connection refused > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:98) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:56) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:57) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:175) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:26:02,102 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-82) Command GetCapabilitiesVDSCommand(HostName = rack, HostId = 1fc56aac-07f1-49db-8972-b5b9567dee77, vds=Host[rack,1fc56aac-07f1-49db-8972-b5b9567dee77]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-09-09 17:26:02,103 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-82) Host rack is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-09 17:26:02,134 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-82) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:26:05,155 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to dell-r210ii-16.rhev.lab.eng.brq.redhat.com/10.34.62.208 >2014-09-09 17:26:05,156 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:26:05,157 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:26:05,157 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-94) Exception during connection: java.util.concurrent.ExecutionException: java.net.ConnectException: Connection refused > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:98) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:56) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:57) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:175) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:26:05,160 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-94) Command GetCapabilitiesVDSCommand(HostName = rack, HostId = 1fc56aac-07f1-49db-8972-b5b9567dee77, vds=Host[rack,1fc56aac-07f1-49db-8972-b5b9567dee77]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-09-09 17:26:05,161 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-94) Host rack is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-09 17:26:05,192 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-94) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:26:08,208 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to dell-r210ii-16.rhev.lab.eng.brq.redhat.com/10.34.62.208 >2014-09-09 17:26:08,208 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:26:08,209 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:26:08,210 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-92) Exception during connection: java.util.concurrent.ExecutionException: java.net.ConnectException: Connection refused > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:98) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:56) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:57) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:175) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:26:08,220 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-92) Command GetCapabilitiesVDSCommand(HostName = rack, HostId = 1fc56aac-07f1-49db-8972-b5b9567dee77, vds=Host[rack,1fc56aac-07f1-49db-8972-b5b9567dee77]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-09-09 17:26:08,220 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-92) Host rack is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-09 17:26:08,253 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-92) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:26:11,270 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to dell-r210ii-16.rhev.lab.eng.brq.redhat.com/10.34.62.208 >2014-09-09 17:26:11,270 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:26:11,271 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:26:11,271 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-3) Exception during connection: java.util.concurrent.ExecutionException: java.net.ConnectException: Connection refused > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:98) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:56) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:57) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:175) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:26:11,273 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-3) Command GetCapabilitiesVDSCommand(HostName = rack, HostId = 1fc56aac-07f1-49db-8972-b5b9567dee77, vds=Host[rack,1fc56aac-07f1-49db-8972-b5b9567dee77]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-09-09 17:26:11,274 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-3) Host rack is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-09 17:26:11,298 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-3) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:26:14,311 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to dell-r210ii-16.rhev.lab.eng.brq.redhat.com/10.34.62.208 >2014-09-09 17:26:14,312 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:26:14,312 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:26:14,313 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-14) Exception during connection: java.util.concurrent.ExecutionException: java.net.ConnectException: Connection refused > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:98) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:56) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:57) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:175) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:26:14,315 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-14) Command GetCapabilitiesVDSCommand(HostName = rack, HostId = 1fc56aac-07f1-49db-8972-b5b9567dee77, vds=Host[rack,1fc56aac-07f1-49db-8972-b5b9567dee77]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-09-09 17:26:14,315 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-14) Host rack is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-09 17:26:14,346 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-14) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:26:17,363 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to dell-r210ii-16.rhev.lab.eng.brq.redhat.com/10.34.62.208 >2014-09-09 17:26:17,363 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:26:17,364 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:26:17,364 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-30) Exception during connection: java.util.concurrent.ExecutionException: java.net.ConnectException: Connection refused > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:98) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:56) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:57) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:175) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:26:17,367 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-30) [5b453678] Command GetCapabilitiesVDSCommand(HostName = rack, HostId = 1fc56aac-07f1-49db-8972-b5b9567dee77, vds=Host[rack,1fc56aac-07f1-49db-8972-b5b9567dee77]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-09-09 17:26:17,367 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-30) [5b453678] Host rack is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-09 17:26:17,409 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-30) [5b453678] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:26:20,424 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to dell-r210ii-16.rhev.lab.eng.brq.redhat.com/10.34.62.208 >2014-09-09 17:26:20,425 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:26:20,425 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:26:20,426 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-26) Exception during connection: java.util.concurrent.ExecutionException: java.net.ConnectException: Connection refused > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:98) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:56) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:57) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:175) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:26:20,429 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-26) Command GetCapabilitiesVDSCommand(HostName = rack, HostId = 1fc56aac-07f1-49db-8972-b5b9567dee77, vds=Host[rack,1fc56aac-07f1-49db-8972-b5b9567dee77]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-09-09 17:26:20,429 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-26) Host rack is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-09 17:26:20,460 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-26) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:26:23,474 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to dell-r210ii-16.rhev.lab.eng.brq.redhat.com/10.34.62.208 >2014-09-09 17:26:23,475 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:26:23,476 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:26:23,476 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-35) Exception during connection: java.util.concurrent.ExecutionException: java.net.ConnectException: Connection refused > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:98) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:56) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:57) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:175) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:26:23,480 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-35) [4d22a9e0] Command GetCapabilitiesVDSCommand(HostName = rack, HostId = 1fc56aac-07f1-49db-8972-b5b9567dee77, vds=Host[rack,1fc56aac-07f1-49db-8972-b5b9567dee77]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-09-09 17:26:23,481 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-35) [4d22a9e0] Host rack is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-09 17:26:23,509 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-35) [4d22a9e0] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:26:26,524 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to dell-r210ii-16.rhev.lab.eng.brq.redhat.com/10.34.62.208 >2014-09-09 17:26:26,525 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:26:26,525 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:26:26,526 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-38) Exception during connection: java.util.concurrent.ExecutionException: java.net.ConnectException: Connection refused > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:98) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:56) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:57) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:175) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:26:26,529 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-38) Command GetCapabilitiesVDSCommand(HostName = rack, HostId = 1fc56aac-07f1-49db-8972-b5b9567dee77, vds=Host[rack,1fc56aac-07f1-49db-8972-b5b9567dee77]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-09-09 17:26:26,529 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-38) Host rack is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-09 17:26:26,569 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-38) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:26:29,585 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to dell-r210ii-16.rhev.lab.eng.brq.redhat.com/10.34.62.208 >2014-09-09 17:26:29,586 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:26:29,587 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:26:29,587 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-46) Exception during connection: java.util.concurrent.ExecutionException: java.net.ConnectException: Connection refused > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:98) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:56) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:57) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:175) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:26:29,590 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-46) Command GetCapabilitiesVDSCommand(HostName = rack, HostId = 1fc56aac-07f1-49db-8972-b5b9567dee77, vds=Host[rack,1fc56aac-07f1-49db-8972-b5b9567dee77]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-09-09 17:26:29,590 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-46) Host rack is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-09 17:26:29,616 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-46) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:26:32,632 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to dell-r210ii-16.rhev.lab.eng.brq.redhat.com/10.34.62.208 >2014-09-09 17:26:32,632 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:26:32,633 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:26:32,634 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-34) Exception during connection: java.util.concurrent.ExecutionException: java.net.ConnectException: Connection refused > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:98) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:56) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:57) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:175) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:26:32,637 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-34) Command GetCapabilitiesVDSCommand(HostName = rack, HostId = 1fc56aac-07f1-49db-8972-b5b9567dee77, vds=Host[rack,1fc56aac-07f1-49db-8972-b5b9567dee77]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-09-09 17:26:32,637 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-34) Host rack is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-09 17:26:32,668 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-34) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:26:35,682 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to dell-r210ii-16.rhev.lab.eng.brq.redhat.com/10.34.62.208 >2014-09-09 17:26:35,683 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:26:35,683 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:26:35,684 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-64) Exception during connection: java.util.concurrent.ExecutionException: java.net.ConnectException: Connection refused > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:98) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:56) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:57) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:175) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:26:35,687 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-64) Command GetCapabilitiesVDSCommand(HostName = rack, HostId = 1fc56aac-07f1-49db-8972-b5b9567dee77, vds=Host[rack,1fc56aac-07f1-49db-8972-b5b9567dee77]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-09-09 17:26:35,687 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-64) Host rack is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-09 17:26:35,733 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-64) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:26:38,748 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to dell-r210ii-16.rhev.lab.eng.brq.redhat.com/10.34.62.208 >2014-09-09 17:26:38,749 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:26:38,749 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:26:38,750 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-55) Exception during connection: java.util.concurrent.ExecutionException: java.net.ConnectException: Connection refused > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:98) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:56) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:57) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:175) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:26:38,752 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-55) Command GetCapabilitiesVDSCommand(HostName = rack, HostId = 1fc56aac-07f1-49db-8972-b5b9567dee77, vds=Host[rack,1fc56aac-07f1-49db-8972-b5b9567dee77]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-09-09 17:26:38,753 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-55) Host rack is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-09 17:26:38,782 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-55) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:26:41,797 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to dell-r210ii-16.rhev.lab.eng.brq.redhat.com/10.34.62.208 >2014-09-09 17:26:41,798 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:26:41,798 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:26:41,799 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-73) Exception during connection: java.util.concurrent.ExecutionException: java.net.ConnectException: Connection refused > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:98) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:56) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:57) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:175) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:26:41,802 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-73) Command GetCapabilitiesVDSCommand(HostName = rack, HostId = 1fc56aac-07f1-49db-8972-b5b9567dee77, vds=Host[rack,1fc56aac-07f1-49db-8972-b5b9567dee77]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-09-09 17:26:41,802 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-73) Host rack is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-09 17:26:41,830 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-73) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:26:44,844 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to dell-r210ii-16.rhev.lab.eng.brq.redhat.com/10.34.62.208 >2014-09-09 17:26:44,845 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:26:44,845 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:26:44,846 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-70) Exception during connection: java.util.concurrent.ExecutionException: java.net.ConnectException: Connection refused > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:98) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:56) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:57) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:175) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:26:44,848 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-70) Command GetCapabilitiesVDSCommand(HostName = rack, HostId = 1fc56aac-07f1-49db-8972-b5b9567dee77, vds=Host[rack,1fc56aac-07f1-49db-8972-b5b9567dee77]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-09-09 17:26:44,848 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-70) Host rack is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-09 17:26:44,878 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-70) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:26:47,892 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to dell-r210ii-16.rhev.lab.eng.brq.redhat.com/10.34.62.208 >2014-09-09 17:26:47,893 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:26:47,893 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:26:47,894 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-82) Exception during connection: java.util.concurrent.ExecutionException: java.net.ConnectException: Connection refused > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:98) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:56) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:57) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:175) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:26:47,896 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-82) Command GetCapabilitiesVDSCommand(HostName = rack, HostId = 1fc56aac-07f1-49db-8972-b5b9567dee77, vds=Host[rack,1fc56aac-07f1-49db-8972-b5b9567dee77]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-09-09 17:26:47,896 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-82) Host rack is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-09 17:26:47,924 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-82) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:26:50,940 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to dell-r210ii-16.rhev.lab.eng.brq.redhat.com/10.34.62.208 >2014-09-09 17:26:50,941 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:26:50,941 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:26:50,942 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-1) Exception during connection: java.util.concurrent.ExecutionException: java.net.ConnectException: Connection refused > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:98) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:56) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:57) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:175) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:26:50,951 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-1) Command GetCapabilitiesVDSCommand(HostName = rack, HostId = 1fc56aac-07f1-49db-8972-b5b9567dee77, vds=Host[rack,1fc56aac-07f1-49db-8972-b5b9567dee77]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-09-09 17:26:50,951 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-1) Host rack is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-09 17:26:50,993 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-1) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:26:54,008 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to dell-r210ii-16.rhev.lab.eng.brq.redhat.com/10.34.62.208 >2014-09-09 17:26:54,009 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:26:54,009 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:26:54,010 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-7) Exception during connection: java.util.concurrent.ExecutionException: java.net.ConnectException: Connection refused > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:98) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:56) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:57) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:175) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:26:54,013 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-7) Command GetCapabilitiesVDSCommand(HostName = rack, HostId = 1fc56aac-07f1-49db-8972-b5b9567dee77, vds=Host[rack,1fc56aac-07f1-49db-8972-b5b9567dee77]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-09-09 17:26:54,013 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-7) Host rack is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-09 17:26:54,037 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-7) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:26:57,052 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to dell-r210ii-16.rhev.lab.eng.brq.redhat.com/10.34.62.208 >2014-09-09 17:26:57,053 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:26:57,053 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:26:57,054 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-13) Exception during connection: java.util.concurrent.ExecutionException: java.net.ConnectException: Connection refused > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:98) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:56) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:57) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:175) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:26:57,057 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-13) Command GetCapabilitiesVDSCommand(HostName = rack, HostId = 1fc56aac-07f1-49db-8972-b5b9567dee77, vds=Host[rack,1fc56aac-07f1-49db-8972-b5b9567dee77]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-09-09 17:26:57,057 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-13) Host rack is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-09 17:26:57,083 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-13) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:27:00,107 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to dell-r210ii-16.rhev.lab.eng.brq.redhat.com/10.34.62.208 >2014-09-09 17:27:00,108 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:27:00,108 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:27:00,109 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-21) Exception during connection: java.util.concurrent.ExecutionException: java.net.ConnectException: Connection refused > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:98) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:56) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:57) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:175) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:27:00,116 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-21) Command GetCapabilitiesVDSCommand(HostName = rack, HostId = 1fc56aac-07f1-49db-8972-b5b9567dee77, vds=Host[rack,1fc56aac-07f1-49db-8972-b5b9567dee77]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-09-09 17:27:00,116 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-21) Host rack is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-09 17:27:00,173 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-21) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:27:03,188 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to dell-r210ii-16.rhev.lab.eng.brq.redhat.com/10.34.62.208 >2014-09-09 17:27:03,189 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:27:03,190 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:27:03,190 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-9) Exception during connection: java.util.concurrent.ExecutionException: java.net.ConnectException: Connection refused > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:98) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:56) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:57) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:175) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:27:03,193 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-9) Command GetCapabilitiesVDSCommand(HostName = rack, HostId = 1fc56aac-07f1-49db-8972-b5b9567dee77, vds=Host[rack,1fc56aac-07f1-49db-8972-b5b9567dee77]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-09-09 17:27:03,193 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-9) Host rack is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-09 17:27:03,222 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-9) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:27:06,237 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to dell-r210ii-16.rhev.lab.eng.brq.redhat.com/10.34.62.208 >2014-09-09 17:27:06,238 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:27:06,238 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:27:06,239 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-19) Exception during connection: java.util.concurrent.ExecutionException: java.net.ConnectException: Connection refused > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:98) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:56) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:57) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:175) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:27:06,242 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-19) Command GetCapabilitiesVDSCommand(HostName = rack, HostId = 1fc56aac-07f1-49db-8972-b5b9567dee77, vds=Host[rack,1fc56aac-07f1-49db-8972-b5b9567dee77]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-09-09 17:27:06,244 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-19) Host rack is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-09 17:27:06,272 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-19) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:27:09,286 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to dell-r210ii-16.rhev.lab.eng.brq.redhat.com/10.34.62.208 >2014-09-09 17:27:09,287 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:27:09,288 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:27:09,288 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-35) Exception during connection: java.util.concurrent.ExecutionException: java.net.ConnectException: Connection refused > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:98) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:56) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:57) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:175) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:27:09,291 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-35) [4d22a9e0] Command GetCapabilitiesVDSCommand(HostName = rack, HostId = 1fc56aac-07f1-49db-8972-b5b9567dee77, vds=Host[rack,1fc56aac-07f1-49db-8972-b5b9567dee77]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-09-09 17:27:09,291 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-35) [4d22a9e0] Host rack is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-09 17:27:09,324 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-35) [4d22a9e0] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:27:12,338 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to dell-r210ii-16.rhev.lab.eng.brq.redhat.com/10.34.62.208 >2014-09-09 17:27:12,339 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:27:12,339 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:27:12,340 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-38) Exception during connection: java.util.concurrent.ExecutionException: java.net.ConnectException: Connection refused > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:98) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:56) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:57) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:175) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:27:12,345 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-38) Command GetCapabilitiesVDSCommand(HostName = rack, HostId = 1fc56aac-07f1-49db-8972-b5b9567dee77, vds=Host[rack,1fc56aac-07f1-49db-8972-b5b9567dee77]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-09-09 17:27:12,346 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-38) Host rack is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-09 17:27:12,375 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-38) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:27:15,389 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to dell-r210ii-16.rhev.lab.eng.brq.redhat.com/10.34.62.208 >2014-09-09 17:27:15,390 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:27:15,390 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:27:15,390 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-54) Exception during connection: java.util.concurrent.ExecutionException: java.net.ConnectException: Connection refused > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:98) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:56) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:57) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:175) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:27:15,393 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-54) Command GetCapabilitiesVDSCommand(HostName = rack, HostId = 1fc56aac-07f1-49db-8972-b5b9567dee77, vds=Host[rack,1fc56aac-07f1-49db-8972-b5b9567dee77]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-09-09 17:27:15,393 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-54) Host rack is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-09 17:27:15,425 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-54) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:27:18,441 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to dell-r210ii-16.rhev.lab.eng.brq.redhat.com/10.34.62.208 >2014-09-09 17:27:18,442 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:27:18,443 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:27:18,443 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-57) Exception during connection: java.util.concurrent.ExecutionException: java.net.ConnectException: Connection refused > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:98) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:56) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:57) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:175) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:27:18,446 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-57) Command GetCapabilitiesVDSCommand(HostName = rack, HostId = 1fc56aac-07f1-49db-8972-b5b9567dee77, vds=Host[rack,1fc56aac-07f1-49db-8972-b5b9567dee77]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-09-09 17:27:18,450 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-57) Host rack is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-09 17:27:18,480 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-57) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:27:21,500 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to dell-r210ii-16.rhev.lab.eng.brq.redhat.com/10.34.62.208 >2014-09-09 17:27:21,501 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:27:21,502 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:27:21,502 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-67) Exception during connection: java.util.concurrent.ExecutionException: java.net.ConnectException: Connection refused > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:98) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:56) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:57) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:175) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:27:21,505 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-67) Command GetCapabilitiesVDSCommand(HostName = rack, HostId = 1fc56aac-07f1-49db-8972-b5b9567dee77, vds=Host[rack,1fc56aac-07f1-49db-8972-b5b9567dee77]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-09-09 17:27:21,510 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-67) Host rack is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-09 17:27:21,548 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-67) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:27:24,564 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to dell-r210ii-16.rhev.lab.eng.brq.redhat.com/10.34.62.208 >2014-09-09 17:27:24,564 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:27:24,565 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:27:24,565 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-78) Exception during connection: java.util.concurrent.ExecutionException: java.net.ConnectException: Connection refused > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:98) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:56) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:57) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:175) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:27:24,574 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-78) Command GetCapabilitiesVDSCommand(HostName = rack, HostId = 1fc56aac-07f1-49db-8972-b5b9567dee77, vds=Host[rack,1fc56aac-07f1-49db-8972-b5b9567dee77]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-09-09 17:27:24,574 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-78) Host rack is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-09 17:27:24,599 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-78) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:27:27,613 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to dell-r210ii-16.rhev.lab.eng.brq.redhat.com/10.34.62.208 >2014-09-09 17:27:27,614 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:27:27,615 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:27:27,615 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-80) Exception during connection: java.util.concurrent.ExecutionException: java.net.ConnectException: Connection refused > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:98) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:56) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:57) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:175) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:27:27,618 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-80) Command GetCapabilitiesVDSCommand(HostName = rack, HostId = 1fc56aac-07f1-49db-8972-b5b9567dee77, vds=Host[rack,1fc56aac-07f1-49db-8972-b5b9567dee77]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-09-09 17:27:27,618 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-80) Host rack is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-09 17:27:27,643 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-80) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:27:30,657 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to dell-r210ii-16.rhev.lab.eng.brq.redhat.com/10.34.62.208 >2014-09-09 17:27:30,658 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:27:30,658 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:27:30,659 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-89) Exception during connection: java.util.concurrent.ExecutionException: java.net.ConnectException: Connection refused > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:98) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:56) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:57) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:175) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:27:30,662 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-89) [55b965af] Command GetCapabilitiesVDSCommand(HostName = rack, HostId = 1fc56aac-07f1-49db-8972-b5b9567dee77, vds=Host[rack,1fc56aac-07f1-49db-8972-b5b9567dee77]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-09-09 17:27:30,662 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-89) [55b965af] Host rack is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-09 17:27:30,688 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-89) [55b965af] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:27:33,703 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to dell-r210ii-16.rhev.lab.eng.brq.redhat.com/10.34.62.208 >2014-09-09 17:27:33,704 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:27:33,704 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:27:33,705 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-99) Exception during connection: java.util.concurrent.ExecutionException: java.net.ConnectException: Connection refused > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:98) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:56) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:57) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:175) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:27:33,708 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-99) Command GetCapabilitiesVDSCommand(HostName = rack, HostId = 1fc56aac-07f1-49db-8972-b5b9567dee77, vds=Host[rack,1fc56aac-07f1-49db-8972-b5b9567dee77]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-09-09 17:27:33,708 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-99) Host rack is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-09 17:27:33,742 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-99) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:27:36,758 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to dell-r210ii-16.rhev.lab.eng.brq.redhat.com/10.34.62.208 >2014-09-09 17:28:39,758 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection timed out > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:28:39,758 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection timed out > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:28:39,759 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-16) Exception during connection: java.util.concurrent.ExecutionException: java.net.ConnectException: Connection timed out > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:98) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:56) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:57) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:175) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection timed out > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:28:39,762 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-16) Command GetCapabilitiesVDSCommand(HostName = rack, HostId = 1fc56aac-07f1-49db-8972-b5b9567dee77, vds=Host[rack,1fc56aac-07f1-49db-8972-b5b9567dee77]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection timed out >2014-09-09 17:28:39,762 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-16) Host rack is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-09 17:28:39,778 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-16) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host rack is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-09 17:28:39,803 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-16) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection timed out > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:28:42,821 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to dell-r210ii-16.rhev.lab.eng.brq.redhat.com/10.34.62.208 >2014-09-09 17:28:45,823 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:28:45,823 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:28:45,824 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-22) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:98) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:56) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:57) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:175) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:28:45,826 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-22) Command GetCapabilitiesVDSCommand(HostName = rack, HostId = 1fc56aac-07f1-49db-8972-b5b9567dee77, vds=Host[rack,1fc56aac-07f1-49db-8972-b5b9567dee77]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-09-09 17:28:45,827 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-22) Host rack is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-09 17:28:45,866 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-22) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:28:48,883 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to dell-r210ii-16.rhev.lab.eng.brq.redhat.com/10.34.62.208 >2014-09-09 17:28:51,885 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:28:51,885 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:28:51,886 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-43) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:98) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:56) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:57) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:175) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:28:51,889 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-43) Command GetCapabilitiesVDSCommand(HostName = rack, HostId = 1fc56aac-07f1-49db-8972-b5b9567dee77, vds=Host[rack,1fc56aac-07f1-49db-8972-b5b9567dee77]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-09-09 17:28:51,889 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-43) Host rack is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-09 17:28:51,920 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-43) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:28:54,937 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to dell-r210ii-16.rhev.lab.eng.brq.redhat.com/10.34.62.208 >2014-09-09 17:28:57,939 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:28:57,939 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:28:57,940 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-63) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:98) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:56) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:57) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:175) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:28:57,942 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-63) Command GetCapabilitiesVDSCommand(HostName = rack, HostId = 1fc56aac-07f1-49db-8972-b5b9567dee77, vds=Host[rack,1fc56aac-07f1-49db-8972-b5b9567dee77]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-09-09 17:28:57,942 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-63) Host rack is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-09 17:28:57,974 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-63) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:29:00,988 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to dell-r210ii-16.rhev.lab.eng.brq.redhat.com/10.34.62.208 >2014-09-09 17:29:03,989 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:29:03,989 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:29:03,990 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-74) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:98) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:56) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:57) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:175) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:29:03,993 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-74) Command GetCapabilitiesVDSCommand(HostName = rack, HostId = 1fc56aac-07f1-49db-8972-b5b9567dee77, vds=Host[rack,1fc56aac-07f1-49db-8972-b5b9567dee77]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-09-09 17:29:03,995 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-74) Host rack is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-09 17:29:04,030 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-74) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:29:07,046 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to dell-r210ii-16.rhev.lab.eng.brq.redhat.com/10.34.62.208 >2014-09-09 17:29:10,047 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:29:10,047 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:29:10,048 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-88) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:98) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:56) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:57) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:175) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:29:10,050 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-88) Command GetCapabilitiesVDSCommand(HostName = rack, HostId = 1fc56aac-07f1-49db-8972-b5b9567dee77, vds=Host[rack,1fc56aac-07f1-49db-8972-b5b9567dee77]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-09-09 17:29:10,051 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-88) Host rack is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-09 17:29:10,083 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-88) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:29:13,098 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to dell-r210ii-16.rhev.lab.eng.brq.redhat.com/10.34.62.208 >2014-09-09 17:29:16,100 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:29:16,100 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:29:16,101 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-94) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:98) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:56) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:57) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:175) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:29:16,103 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-94) Command GetCapabilitiesVDSCommand(HostName = rack, HostId = 1fc56aac-07f1-49db-8972-b5b9567dee77, vds=Host[rack,1fc56aac-07f1-49db-8972-b5b9567dee77]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-09-09 17:29:16,104 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-94) Host rack is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-09 17:29:16,136 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-94) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:29:19,154 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to dell-r210ii-16.rhev.lab.eng.brq.redhat.com/10.34.62.208 >2014-09-09 17:29:22,155 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:29:22,155 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:29:22,156 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-7) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:98) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:56) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:57) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:175) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:29:22,159 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-7) Command GetCapabilitiesVDSCommand(HostName = rack, HostId = 1fc56aac-07f1-49db-8972-b5b9567dee77, vds=Host[rack,1fc56aac-07f1-49db-8972-b5b9567dee77]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-09-09 17:29:22,159 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-7) Host rack is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-09 17:29:22,187 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-7) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:29:25,202 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to dell-r210ii-16.rhev.lab.eng.brq.redhat.com/10.34.62.208 >2014-09-09 17:29:28,203 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:29:28,203 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:29:28,204 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-16) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:98) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:56) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:57) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:175) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:29:28,206 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-16) Command GetCapabilitiesVDSCommand(HostName = rack, HostId = 1fc56aac-07f1-49db-8972-b5b9567dee77, vds=Host[rack,1fc56aac-07f1-49db-8972-b5b9567dee77]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-09-09 17:29:28,206 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-16) Host rack is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-09 17:29:28,247 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-16) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:29:31,263 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to dell-r210ii-16.rhev.lab.eng.brq.redhat.com/10.34.62.208 >2014-09-09 17:29:49,265 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:29:49,265 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:29:49,266 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-31) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:98) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:56) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:57) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:175) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:29:49,271 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-31) Command GetCapabilitiesVDSCommand(HostName = rack, HostId = 1fc56aac-07f1-49db-8972-b5b9567dee77, vds=Host[rack,1fc56aac-07f1-49db-8972-b5b9567dee77]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-09-09 17:29:49,271 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-31) Host rack is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-09 17:29:49,304 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-31) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:29:52,320 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to dell-r210ii-16.rhev.lab.eng.brq.redhat.com/10.34.62.208 >2014-09-09 17:29:55,322 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:29:55,323 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:29:55,323 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-73) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:98) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:56) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:57) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:175) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:29:55,328 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-73) Command GetCapabilitiesVDSCommand(HostName = rack, HostId = 1fc56aac-07f1-49db-8972-b5b9567dee77, vds=Host[rack,1fc56aac-07f1-49db-8972-b5b9567dee77]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-09-09 17:29:55,328 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-73) Host rack is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-09 17:29:55,362 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-73) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:29:58,381 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to dell-r210ii-16.rhev.lab.eng.brq.redhat.com/10.34.62.208 >2014-09-09 17:30:01,382 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:30:01,382 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:30:01,383 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-90) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:98) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:56) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:57) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:175) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:30:01,386 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-90) [18a5c631] Command GetCapabilitiesVDSCommand(HostName = rack, HostId = 1fc56aac-07f1-49db-8972-b5b9567dee77, vds=Host[rack,1fc56aac-07f1-49db-8972-b5b9567dee77]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-09-09 17:30:01,386 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-90) [18a5c631] Host rack is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-09 17:30:01,416 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-90) [18a5c631] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:30:04,434 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to dell-r210ii-16.rhev.lab.eng.brq.redhat.com/10.34.62.208 >2014-09-09 17:30:07,436 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:30:07,436 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:30:07,437 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-94) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:98) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:56) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:57) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:175) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:30:07,440 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-94) Command GetCapabilitiesVDSCommand(HostName = rack, HostId = 1fc56aac-07f1-49db-8972-b5b9567dee77, vds=Host[rack,1fc56aac-07f1-49db-8972-b5b9567dee77]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-09-09 17:30:07,442 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-94) Host rack is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-09 17:30:07,472 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-94) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:30:10,486 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to dell-r210ii-16.rhev.lab.eng.brq.redhat.com/10.34.62.208 >2014-09-09 17:30:13,488 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:30:13,488 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:30:13,489 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-14) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:98) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:56) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:57) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:175) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:30:13,491 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-14) Command GetCapabilitiesVDSCommand(HostName = rack, HostId = 1fc56aac-07f1-49db-8972-b5b9567dee77, vds=Host[rack,1fc56aac-07f1-49db-8972-b5b9567dee77]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-09-09 17:30:13,491 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-14) Host rack is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-09 17:30:13,515 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-14) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:30:16,528 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to dell-r210ii-16.rhev.lab.eng.brq.redhat.com/10.34.62.208 >2014-09-09 17:30:19,529 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:30:19,529 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:30:19,530 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-16) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:98) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:56) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:57) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:175) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:30:19,533 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-16) Command GetCapabilitiesVDSCommand(HostName = rack, HostId = 1fc56aac-07f1-49db-8972-b5b9567dee77, vds=Host[rack,1fc56aac-07f1-49db-8972-b5b9567dee77]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-09-09 17:30:19,533 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-16) Host rack is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-09 17:30:19,565 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-16) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:30:22,580 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to dell-r210ii-16.rhev.lab.eng.brq.redhat.com/10.34.62.208 >2014-09-09 17:30:25,582 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:30:25,582 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:30:25,583 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-44) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:98) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:56) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:57) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:175) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:30:25,585 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-44) Command GetCapabilitiesVDSCommand(HostName = rack, HostId = 1fc56aac-07f1-49db-8972-b5b9567dee77, vds=Host[rack,1fc56aac-07f1-49db-8972-b5b9567dee77]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-09-09 17:30:25,585 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-44) Host rack is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-09 17:30:25,613 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-44) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:30:28,628 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to dell-r210ii-16.rhev.lab.eng.brq.redhat.com/10.34.62.208 >2014-09-09 17:30:31,629 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:30:31,630 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:30:31,630 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-52) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:98) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:56) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:57) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:175) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:30:31,633 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-52) Command GetCapabilitiesVDSCommand(HostName = rack, HostId = 1fc56aac-07f1-49db-8972-b5b9567dee77, vds=Host[rack,1fc56aac-07f1-49db-8972-b5b9567dee77]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-09-09 17:30:31,633 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-52) Host rack is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-09 17:30:31,662 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-52) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:30:34,677 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to dell-r210ii-16.rhev.lab.eng.brq.redhat.com/10.34.62.208 >2014-09-09 17:30:37,678 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:30:37,678 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:30:37,679 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-60) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:98) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:56) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:57) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:175) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:30:37,682 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-60) Command GetCapabilitiesVDSCommand(HostName = rack, HostId = 1fc56aac-07f1-49db-8972-b5b9567dee77, vds=Host[rack,1fc56aac-07f1-49db-8972-b5b9567dee77]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-09-09 17:30:37,682 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-60) Host rack is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-09 17:30:37,718 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-60) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:30:40,732 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to dell-r210ii-16.rhev.lab.eng.brq.redhat.com/10.34.62.208 >2014-09-09 17:30:43,734 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:30:43,734 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:30:43,735 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-71) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:98) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:56) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:57) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:175) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:30:43,738 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-71) Command GetCapabilitiesVDSCommand(HostName = rack, HostId = 1fc56aac-07f1-49db-8972-b5b9567dee77, vds=Host[rack,1fc56aac-07f1-49db-8972-b5b9567dee77]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-09-09 17:30:43,738 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-71) Host rack is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-09 17:30:43,764 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-71) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:30:46,779 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to dell-r210ii-16.rhev.lab.eng.brq.redhat.com/10.34.62.208 >2014-09-09 17:30:48,780 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:30:48,781 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:30:48,782 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-36) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:98) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:56) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:57) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:175) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:30:48,785 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-36) Command GetCapabilitiesVDSCommand(HostName = rack, HostId = 1fc56aac-07f1-49db-8972-b5b9567dee77, vds=Host[rack,1fc56aac-07f1-49db-8972-b5b9567dee77]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-09-09 17:30:48,785 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-36) Host rack is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-09 17:30:48,815 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-36) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:30:51,830 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to dell-r210ii-16.rhev.lab.eng.brq.redhat.com/10.34.62.208 >2014-09-09 17:30:51,830 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:30:51,831 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:30:51,831 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-92) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:98) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:56) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:57) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:175) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:30:51,834 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-92) Command GetCapabilitiesVDSCommand(HostName = rack, HostId = 1fc56aac-07f1-49db-8972-b5b9567dee77, vds=Host[rack,1fc56aac-07f1-49db-8972-b5b9567dee77]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-09-09 17:30:51,834 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-92) Host rack is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-09 17:30:51,864 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-92) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:30:54,881 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to dell-r210ii-16.rhev.lab.eng.brq.redhat.com/10.34.62.208 >2014-09-09 17:30:54,882 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:30:54,882 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:30:54,882 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-4) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:98) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:56) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:57) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:175) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:30:54,885 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-4) Command GetCapabilitiesVDSCommand(HostName = rack, HostId = 1fc56aac-07f1-49db-8972-b5b9567dee77, vds=Host[rack,1fc56aac-07f1-49db-8972-b5b9567dee77]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-09-09 17:30:54,885 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-4) Host rack is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-09 17:30:54,911 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-4) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:30:57,929 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to dell-r210ii-16.rhev.lab.eng.brq.redhat.com/10.34.62.208 >2014-09-09 17:30:57,930 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:30:57,930 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:30:57,931 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-18) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:98) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:56) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:57) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:175) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:30:57,942 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-18) Command GetCapabilitiesVDSCommand(HostName = rack, HostId = 1fc56aac-07f1-49db-8972-b5b9567dee77, vds=Host[rack,1fc56aac-07f1-49db-8972-b5b9567dee77]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-09-09 17:30:57,942 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-18) Host rack is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-09 17:30:57,981 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-18) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:31:00,995 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to dell-r210ii-16.rhev.lab.eng.brq.redhat.com/10.34.62.208 >2014-09-09 17:31:00,995 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:31:00,996 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:31:00,996 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-24) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:98) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:56) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:57) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:175) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:31:00,999 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-24) Command GetCapabilitiesVDSCommand(HostName = rack, HostId = 1fc56aac-07f1-49db-8972-b5b9567dee77, vds=Host[rack,1fc56aac-07f1-49db-8972-b5b9567dee77]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-09-09 17:31:00,999 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-24) Host rack is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-09 17:31:01,027 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-24) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:31:04,043 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to dell-r210ii-16.rhev.lab.eng.brq.redhat.com/10.34.62.208 >2014-09-09 17:31:04,044 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:31:04,044 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:31:04,045 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-16) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:98) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:56) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:57) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:175) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:31:04,054 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-16) Command GetCapabilitiesVDSCommand(HostName = rack, HostId = 1fc56aac-07f1-49db-8972-b5b9567dee77, vds=Host[rack,1fc56aac-07f1-49db-8972-b5b9567dee77]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-09-09 17:31:04,055 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-16) Host rack is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-09 17:31:04,086 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-16) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:31:07,101 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to dell-r210ii-16.rhev.lab.eng.brq.redhat.com/10.34.62.208 >2014-09-09 17:31:07,102 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:31:07,102 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:31:07,102 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-2) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:98) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:56) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:57) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:175) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:31:07,110 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-2) Command GetCapabilitiesVDSCommand(HostName = rack, HostId = 1fc56aac-07f1-49db-8972-b5b9567dee77, vds=Host[rack,1fc56aac-07f1-49db-8972-b5b9567dee77]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-09-09 17:31:07,111 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-2) Host rack is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-09 17:31:07,139 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-2) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:31:10,153 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to dell-r210ii-16.rhev.lab.eng.brq.redhat.com/10.34.62.208 >2014-09-09 17:31:10,154 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:31:10,154 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:31:10,155 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-38) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:98) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:56) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:57) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:175) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:31:10,157 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-38) Command GetCapabilitiesVDSCommand(HostName = rack, HostId = 1fc56aac-07f1-49db-8972-b5b9567dee77, vds=Host[rack,1fc56aac-07f1-49db-8972-b5b9567dee77]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-09-09 17:31:10,157 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-38) Host rack is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-09 17:31:10,196 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-38) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:31:13,216 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to dell-r210ii-16.rhev.lab.eng.brq.redhat.com/10.34.62.208 >2014-09-09 17:31:13,217 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:31:13,217 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:31:13,218 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-43) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:98) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:56) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:57) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:175) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:31:13,221 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-43) Command GetCapabilitiesVDSCommand(HostName = rack, HostId = 1fc56aac-07f1-49db-8972-b5b9567dee77, vds=Host[rack,1fc56aac-07f1-49db-8972-b5b9567dee77]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-09-09 17:31:13,222 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-43) Host rack is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-09 17:31:13,249 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-43) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:31:16,263 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to dell-r210ii-16.rhev.lab.eng.brq.redhat.com/10.34.62.208 >2014-09-09 17:31:16,264 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:31:16,264 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:31:16,265 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-57) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:98) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:56) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:57) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:175) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:31:16,269 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-57) Command GetCapabilitiesVDSCommand(HostName = rack, HostId = 1fc56aac-07f1-49db-8972-b5b9567dee77, vds=Host[rack,1fc56aac-07f1-49db-8972-b5b9567dee77]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-09-09 17:31:16,269 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-57) Host rack is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-09 17:31:16,297 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-57) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:31:19,310 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to dell-r210ii-16.rhev.lab.eng.brq.redhat.com/10.34.62.208 >2014-09-09 17:31:19,311 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:31:19,312 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:31:19,312 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-50) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:98) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:56) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:57) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:175) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:31:19,314 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-50) Command GetCapabilitiesVDSCommand(HostName = rack, HostId = 1fc56aac-07f1-49db-8972-b5b9567dee77, vds=Host[rack,1fc56aac-07f1-49db-8972-b5b9567dee77]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-09-09 17:31:19,315 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-50) Host rack is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-09 17:31:19,345 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-50) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:31:22,365 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to dell-r210ii-16.rhev.lab.eng.brq.redhat.com/10.34.62.208 >2014-09-09 17:31:22,366 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:31:22,366 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:31:22,367 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-65) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:98) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:56) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:57) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:175) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:31:22,369 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-65) Command GetCapabilitiesVDSCommand(HostName = rack, HostId = 1fc56aac-07f1-49db-8972-b5b9567dee77, vds=Host[rack,1fc56aac-07f1-49db-8972-b5b9567dee77]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-09-09 17:31:22,369 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-65) Host rack is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-09 17:31:22,405 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-65) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:31:25,424 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to dell-r210ii-16.rhev.lab.eng.brq.redhat.com/10.34.62.208 >2014-09-09 17:31:25,424 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:31:25,425 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:31:25,425 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-80) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:98) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:56) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:57) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:175) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:31:25,428 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-80) Command GetCapabilitiesVDSCommand(HostName = rack, HostId = 1fc56aac-07f1-49db-8972-b5b9567dee77, vds=Host[rack,1fc56aac-07f1-49db-8972-b5b9567dee77]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-09-09 17:31:25,428 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-80) Host rack is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-09 17:31:25,455 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-80) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:31:28,474 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to dell-r210ii-16.rhev.lab.eng.brq.redhat.com/10.34.62.208 >2014-09-09 17:31:28,478 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:31:28,478 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:31:28,478 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-70) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:98) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:56) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:57) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:175) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:31:28,481 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-70) Command GetCapabilitiesVDSCommand(HostName = rack, HostId = 1fc56aac-07f1-49db-8972-b5b9567dee77, vds=Host[rack,1fc56aac-07f1-49db-8972-b5b9567dee77]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-09-09 17:31:28,481 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-70) Host rack is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-09 17:31:28,511 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-70) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:31:31,526 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to dell-r210ii-16.rhev.lab.eng.brq.redhat.com/10.34.62.208 >2014-09-09 17:31:31,526 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:31:31,527 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:31:31,527 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-88) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:98) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:56) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:57) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:175) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:31:31,530 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-88) Command GetCapabilitiesVDSCommand(HostName = rack, HostId = 1fc56aac-07f1-49db-8972-b5b9567dee77, vds=Host[rack,1fc56aac-07f1-49db-8972-b5b9567dee77]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-09-09 17:31:31,530 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-88) Host rack is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-09 17:31:31,556 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-88) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:31:34,571 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to dell-r210ii-16.rhev.lab.eng.brq.redhat.com/10.34.62.208 >2014-09-09 17:31:34,572 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:31:34,572 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:31:34,573 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-90) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:98) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:56) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:57) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:175) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:31:34,576 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-90) [18a5c631] Command GetCapabilitiesVDSCommand(HostName = rack, HostId = 1fc56aac-07f1-49db-8972-b5b9567dee77, vds=Host[rack,1fc56aac-07f1-49db-8972-b5b9567dee77]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-09-09 17:31:34,576 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-90) [18a5c631] Host rack is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-09 17:31:34,606 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-90) [18a5c631] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:31:37,620 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to dell-r210ii-16.rhev.lab.eng.brq.redhat.com/10.34.62.208 >2014-09-09 17:31:37,621 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:31:37,622 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:31:37,622 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-97) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:98) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:56) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:57) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:175) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:31:37,624 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-97) Command GetCapabilitiesVDSCommand(HostName = rack, HostId = 1fc56aac-07f1-49db-8972-b5b9567dee77, vds=Host[rack,1fc56aac-07f1-49db-8972-b5b9567dee77]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-09-09 17:31:37,624 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-97) Host rack is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-09 17:31:37,657 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-97) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:31:40,672 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to dell-r210ii-16.rhev.lab.eng.brq.redhat.com/10.34.62.208 >2014-09-09 17:31:40,673 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:31:40,673 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:31:40,674 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-12) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:98) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:56) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:57) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:175) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:31:40,677 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-12) Command GetCapabilitiesVDSCommand(HostName = rack, HostId = 1fc56aac-07f1-49db-8972-b5b9567dee77, vds=Host[rack,1fc56aac-07f1-49db-8972-b5b9567dee77]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-09-09 17:31:40,677 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-12) Host rack is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-09 17:31:40,694 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-12) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host rack is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-09 17:31:40,719 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-12) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:31:43,736 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to dell-r210ii-16.rhev.lab.eng.brq.redhat.com/10.34.62.208 >2014-09-09 17:31:43,737 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:31:43,737 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:31:43,738 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-18) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:98) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:56) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:57) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:175) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:31:43,741 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-18) Command GetCapabilitiesVDSCommand(HostName = rack, HostId = 1fc56aac-07f1-49db-8972-b5b9567dee77, vds=Host[rack,1fc56aac-07f1-49db-8972-b5b9567dee77]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-09-09 17:31:43,745 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-18) Host rack is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-09 17:31:43,774 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-18) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:31:46,789 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to dell-r210ii-16.rhev.lab.eng.brq.redhat.com/10.34.62.208 >2014-09-09 17:31:46,793 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:31:46,794 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:31:46,795 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-15) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:98) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:56) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:57) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:175) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:31:46,797 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-15) Command GetCapabilitiesVDSCommand(HostName = rack, HostId = 1fc56aac-07f1-49db-8972-b5b9567dee77, vds=Host[rack,1fc56aac-07f1-49db-8972-b5b9567dee77]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-09-09 17:31:46,798 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-15) Host rack is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-09 17:31:46,832 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-15) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:31:49,848 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to dell-r210ii-16.rhev.lab.eng.brq.redhat.com/10.34.62.208 >2014-09-09 17:31:49,848 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:31:49,849 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:31:49,849 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-25) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:98) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:56) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:57) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:175) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:31:49,863 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-25) [27472998] Command GetCapabilitiesVDSCommand(HostName = rack, HostId = 1fc56aac-07f1-49db-8972-b5b9567dee77, vds=Host[rack,1fc56aac-07f1-49db-8972-b5b9567dee77]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-09-09 17:31:49,863 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-25) [27472998] Host rack is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-09 17:31:49,887 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-25) [27472998] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:31:52,904 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to dell-r210ii-16.rhev.lab.eng.brq.redhat.com/10.34.62.208 >2014-09-09 17:31:52,904 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:31:52,905 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:31:52,905 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-39) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:98) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:56) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:57) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:175) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:31:52,908 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-39) Command GetCapabilitiesVDSCommand(HostName = rack, HostId = 1fc56aac-07f1-49db-8972-b5b9567dee77, vds=Host[rack,1fc56aac-07f1-49db-8972-b5b9567dee77]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-09-09 17:31:52,908 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-39) Host rack is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-09 17:31:52,939 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-39) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:31:55,954 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to dell-r210ii-16.rhev.lab.eng.brq.redhat.com/10.34.62.208 >2014-09-09 17:31:55,954 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:31:55,955 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:31:55,955 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-53) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:98) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:56) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:57) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:175) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:31:55,958 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-53) [5733dd3e] Command GetCapabilitiesVDSCommand(HostName = rack, HostId = 1fc56aac-07f1-49db-8972-b5b9567dee77, vds=Host[rack,1fc56aac-07f1-49db-8972-b5b9567dee77]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-09-09 17:31:55,959 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-53) [5733dd3e] Host rack is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-09 17:31:55,987 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-53) [5733dd3e] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:31:59,003 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to dell-r210ii-16.rhev.lab.eng.brq.redhat.com/10.34.62.208 >2014-09-09 17:31:59,004 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:31:59,031 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:31:59,032 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-54) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:98) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:56) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:57) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:175) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:31:59,034 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-54) Command GetCapabilitiesVDSCommand(HostName = rack, HostId = 1fc56aac-07f1-49db-8972-b5b9567dee77, vds=Host[rack,1fc56aac-07f1-49db-8972-b5b9567dee77]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-09-09 17:31:59,034 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-54) Host rack is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-09 17:31:59,061 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-54) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:32:02,075 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to dell-r210ii-16.rhev.lab.eng.brq.redhat.com/10.34.62.208 >2014-09-09 17:32:02,076 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:32:02,076 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:32:02,076 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-68) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:98) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:56) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:57) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:175) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:32:02,078 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-68) Command GetCapabilitiesVDSCommand(HostName = rack, HostId = 1fc56aac-07f1-49db-8972-b5b9567dee77, vds=Host[rack,1fc56aac-07f1-49db-8972-b5b9567dee77]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-09-09 17:32:02,079 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-68) Host rack is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-09 17:32:02,107 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-68) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:32:05,035 INFO [org.ovirt.engine.core.bll.MaintenanceNumberOfVdssCommand] (org.ovirt.thread.pool-6-thread-33) [64ef3315] Running command: MaintenanceNumberOfVdssCommand internal: false. Entities affected : ID: 1fc56aac-07f1-49db-8972-b5b9567dee77 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2014-09-09 17:32:05,122 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to dell-r210ii-16.rhev.lab.eng.brq.redhat.com/10.34.62.208 >2014-09-09 17:32:05,122 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:32:05,123 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:32:05,125 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-65) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:98) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:56) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:57) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:175) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:32:05,128 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-65) Command GetCapabilitiesVDSCommand(HostName = rack, HostId = 1fc56aac-07f1-49db-8972-b5b9567dee77, vds=Host[rack,1fc56aac-07f1-49db-8972-b5b9567dee77]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-09-09 17:32:05,128 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-65) Host rack is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-09 17:32:05,175 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-65) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:32:05,180 INFO [org.ovirt.engine.core.bll.MaintenanceVdsCommand] (org.ovirt.thread.pool-6-thread-33) [64ef3315] Running command: MaintenanceVdsCommand internal: true. Entities affected : ID: 1fc56aac-07f1-49db-8972-b5b9567dee77 Type: VDS >2014-09-09 17:32:05,194 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-6-thread-33) [64ef3315] START, SetVdsStatusVDSCommand(HostName = rack, HostId = 1fc56aac-07f1-49db-8972-b5b9567dee77, status=Maintenance, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 7fe96046 >2014-09-09 17:32:05,217 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-6-thread-33) [64ef3315] FINISH, SetVdsStatusVDSCommand, log id: 7fe96046 >2014-09-09 17:32:05,229 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-33) [64ef3315] Correlation ID: 64ef3315, Job ID: bf9a2a2d-aa3e-4ec7-afac-34668f6e3c4e, Call Stack: null, Custom Event ID: -1, Message: Host rack was switched to Maintenance mode by admin. >2014-09-09 17:32:17,779 INFO [org.ovirt.engine.core.bll.RemoveVdsCommand] (ajp-/127.0.0.1:8702-3) [2d507abd] Lock Acquired to object EngineLock [exclusiveLocks= key: 1fc56aac-07f1-49db-8972-b5b9567dee77 value: VDS >, sharedLocks= ] >2014-09-09 17:32:17,795 INFO [org.ovirt.engine.core.bll.RemoveVdsCommand] (org.ovirt.thread.pool-6-thread-35) [2d507abd] Running command: RemoveVdsCommand internal: false. Entities affected : ID: 1fc56aac-07f1-49db-8972-b5b9567dee77 Type: VDSAction group DELETE_HOST with role type ADMIN >2014-09-09 17:32:17,826 INFO [org.ovirt.engine.core.vdsbroker.RemoveVdsVDSCommand] (org.ovirt.thread.pool-6-thread-35) [2d507abd] START, RemoveVdsVDSCommand( HostId = 1fc56aac-07f1-49db-8972-b5b9567dee77), log id: 668ff23e >2014-09-09 17:32:17,827 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (org.ovirt.thread.pool-6-thread-35) [2d507abd] vdsManager::disposing >2014-09-09 17:32:17,828 INFO [org.ovirt.engine.core.vdsbroker.RemoveVdsVDSCommand] (org.ovirt.thread.pool-6-thread-35) [2d507abd] FINISH, RemoveVdsVDSCommand, log id: 668ff23e >2014-09-09 17:32:17,836 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-35) [2d507abd] Correlation ID: 2d507abd, Call Stack: null, Custom Event ID: -1, Message: Host rack was removed by admin. >2014-09-09 17:32:17,837 INFO [org.ovirt.engine.core.bll.RemoveVdsCommand] (org.ovirt.thread.pool-6-thread-35) [2d507abd] Lock freed to object EngineLock [exclusiveLocks= key: 1fc56aac-07f1-49db-8972-b5b9567dee77 value: VDS >, sharedLocks= ] >2014-09-09 17:32:33,607 WARN [org.ovirt.engine.core.bll.AddVdsCommand] (ajp-/127.0.0.1:8702-8) [d4f8ee6] Failed to initiate vdsm-id request on host with message Command returned failure code 127 during SSH session 'root@dell-r210ii-16.rhev.lab.eng.brq.redhat.com' >2014-09-09 17:32:33,637 INFO [org.ovirt.engine.core.bll.AddVdsCommand] (ajp-/127.0.0.1:8702-8) [d4f8ee6] Running command: AddVdsCommand internal: false. Entities affected : ID: 00000001-0001-0001-0001-000000000399 Type: VdsGroupsAction group CREATE_HOST with role type ADMIN >2014-09-09 17:32:33,712 INFO [org.ovirt.engine.core.bll.AddVdsSpmIdCommand] (ajp-/127.0.0.1:8702-8) [139b591] Running command: AddVdsSpmIdCommand internal: true. Entities affected : ID: 7ebdb254-efb6-480b-8c4e-a7dbdc759ed4 Type: VDS >2014-09-09 17:32:33,727 INFO [org.ovirt.engine.core.bll.AddVdsSpmIdCommand] (ajp-/127.0.0.1:8702-8) [139b591] Lock freed to object EngineLock [exclusiveLocks= key: 00000002-0002-0002-0002-0000000001ad value: REGISTER_VDS >, sharedLocks= ] >2014-09-09 17:32:33,730 INFO [org.ovirt.engine.core.vdsbroker.RemoveVdsVDSCommand] (ajp-/127.0.0.1:8702-8) [139b591] START, RemoveVdsVDSCommand(HostName = rack, HostId = 7ebdb254-efb6-480b-8c4e-a7dbdc759ed4), log id: 26d724be >2014-09-09 17:32:33,731 INFO [org.ovirt.engine.core.vdsbroker.RemoveVdsVDSCommand] (ajp-/127.0.0.1:8702-8) [139b591] FINISH, RemoveVdsVDSCommand, log id: 26d724be >2014-09-09 17:32:33,733 INFO [org.ovirt.engine.core.vdsbroker.AddVdsVDSCommand] (ajp-/127.0.0.1:8702-8) [139b591] START, AddVdsVDSCommand(HostName = rack, HostId = 7ebdb254-efb6-480b-8c4e-a7dbdc759ed4), log id: d2b5bbd >2014-09-09 17:32:33,733 INFO [org.ovirt.engine.core.vdsbroker.AddVdsVDSCommand] (ajp-/127.0.0.1:8702-8) [139b591] AddVds - entered , starting logic to add VDS 7ebdb254-efb6-480b-8c4e-a7dbdc759ed4 >2014-09-09 17:32:33,736 INFO [org.ovirt.engine.core.vdsbroker.AddVdsVDSCommand] (ajp-/127.0.0.1:8702-8) [139b591] AddVds - VDS 7ebdb254-efb6-480b-8c4e-a7dbdc759ed4 was added, will try to add it to the resource manager >2014-09-09 17:32:33,736 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (ajp-/127.0.0.1:8702-8) [139b591] Entered VdsManager constructor >2014-09-09 17:32:33,749 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (ajp-/127.0.0.1:8702-8) [139b591] Initialize vdsBroker (dell-r210ii-16.rhev.lab.eng.brq.redhat.com,54,321) >2014-09-09 17:32:33,750 INFO [org.ovirt.engine.core.vdsbroker.ResourceManager] (ajp-/127.0.0.1:8702-8) [139b591] VDS 7ebdb254-efb6-480b-8c4e-a7dbdc759ed4 was added to the Resource Manager >2014-09-09 17:32:33,751 INFO [org.ovirt.engine.core.vdsbroker.AddVdsVDSCommand] (ajp-/127.0.0.1:8702-8) [139b591] FINISH, AddVdsVDSCommand, log id: d2b5bbd >2014-09-09 17:32:33,766 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-8) [139b591] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host rack. >2014-09-09 17:32:33,804 INFO [org.ovirt.engine.core.bll.InstallVdsInternalCommand] (org.ovirt.thread.pool-6-thread-36) [6093dd15] Lock Acquired to object EngineLock [exclusiveLocks= key: 7ebdb254-efb6-480b-8c4e-a7dbdc759ed4 value: VDS >, sharedLocks= ] >2014-09-09 17:32:33,805 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-8) [139b591] Correlation ID: d4f8ee6, Job ID: 28ad829d-a61d-4309-a303-4bf2b7ff5d54, Call Stack: null, Custom Event ID: -1, Message: Host rack was added by admin. >2014-09-09 17:32:33,823 INFO [org.ovirt.engine.core.bll.InstallVdsInternalCommand] (org.ovirt.thread.pool-6-thread-36) [6093dd15] Running command: InstallVdsInternalCommand internal: true. Entities affected : ID: 7ebdb254-efb6-480b-8c4e-a7dbdc759ed4 Type: VDS >2014-09-09 17:32:33,823 INFO [org.ovirt.engine.core.bll.InstallVdsInternalCommand] (org.ovirt.thread.pool-6-thread-36) [6093dd15] Before Installation host 7ebdb254-efb6-480b-8c4e-a7dbdc759ed4, rack >2014-09-09 17:32:33,828 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-6-thread-36) [6093dd15] START, SetVdsStatusVDSCommand(HostName = rack, HostId = 7ebdb254-efb6-480b-8c4e-a7dbdc759ed4, status=Installing, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 182d4fcc >2014-09-09 17:32:33,838 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-6-thread-36) [6093dd15] FINISH, SetVdsStatusVDSCommand, log id: 182d4fcc >2014-09-09 17:32:33,875 INFO [org.ovirt.engine.core.bll.InstallerMessages] (org.ovirt.thread.pool-6-thread-36) [6093dd15] Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Connected to host dell-r210ii-16.rhev.lab.eng.brq.redhat.com with SSH key fingerprint: 25:12:7b:de:30:1d:b3:5e:bf:d2:a6:c7:37:77:d3:b1 >2014-09-09 17:32:33,889 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-36) [6093dd15] Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Connected to host dell-r210ii-16.rhev.lab.eng.brq.redhat.com with SSH key fingerprint: 25:12:7b:de:30:1d:b3:5e:bf:d2:a6:c7:37:77:d3:b1. >2014-09-09 17:32:33,954 INFO [org.ovirt.engine.core.bll.VdsDeploy] (org.ovirt.thread.pool-6-thread-36) [6093dd15] Installation of dell-r210ii-16.rhev.lab.eng.brq.redhat.com. Executing command via SSH umask 0077; MYTMP="$(TMPDIR="${OVIRT_TMPDIR}" mktemp -t ovirt-XXXXXXXXXX)"; trap "chmod -R u+rwX \"${MYTMP}\" > /dev/null 2>&1; rm -fr \"${MYTMP}\" > /dev/null 2>&1" 0; rm -fr "${MYTMP}" && mkdir "${MYTMP}" && tar --warning=no-timestamp -C "${MYTMP}" -x && "${MYTMP}"/setup DIALOG/dialect=str:machine DIALOG/customization=bool:True < /var/cache/ovirt-engine/ovirt-host-deploy.tar >2014-09-09 17:32:33,967 INFO [org.ovirt.engine.core.uutils.ssh.SSHDialog] (org.ovirt.thread.pool-6-thread-36) SSH execute root@dell-r210ii-16.rhev.lab.eng.brq.redhat.com 'umask 0077; MYTMP="$(TMPDIR="${OVIRT_TMPDIR}" mktemp -t ovirt-XXXXXXXXXX)"; trap "chmod -R u+rwX \"${MYTMP}\" > /dev/null 2>&1; rm -fr \"${MYTMP}\" > /dev/null 2>&1" 0; rm -fr "${MYTMP}" && mkdir "${MYTMP}" && tar --warning=no-timestamp -C "${MYTMP}" -x && "${MYTMP}"/setup DIALOG/dialect=str:machine DIALOG/customization=bool:True' >2014-09-09 17:32:36,753 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-16) Initializing Host: rack >2014-09-09 17:32:55,661 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum Status: Downloading Packages >2014-09-09 17:32:55,673 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Status: Downloading Packages. >2014-09-09 17:32:56,654 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum Download/Verify: libselinux-python-2.0.94-5.3.el6_4.1.x86_64 >2014-09-09 17:32:56,668 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Download/Verify: libselinux-python-2.0.94-5.3.el6_4.1.x86_64. >2014-09-09 17:32:56,668 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum Status: Check Package Signatures >2014-09-09 17:32:56,679 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Status: Check Package Signatures. >2014-09-09 17:32:56,679 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum Status: Running Test Transaction >2014-09-09 17:32:56,692 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Status: Running Test Transaction. >2014-09-09 17:32:56,800 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum Status: Running Transaction >2014-09-09 17:32:56,811 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Status: Running Transaction. >2014-09-09 17:32:57,295 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum install: 1/1: libselinux-python-2.0.94-5.3.el6_4.1.x86_64 >2014-09-09 17:32:57,307 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum install: 1/1: libselinux-python-2.0.94-5.3.el6_4.1.x86_64. >2014-09-09 17:33:02,641 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum Verify: 1/1: libselinux-python.x86_64 0:2.0.94-5.3.el6_4.1 - u >2014-09-09 17:33:02,691 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Verify: 1/1: libselinux-python.x86_64 0:2.0.94-5.3.el6_4.1 - u. >2014-09-09 17:33:09,574 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Stage: Initializing >2014-09-09 17:33:09,586 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Stage: Initializing. >2014-09-09 17:33:09,600 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Stage: Environment setup >2014-09-09 17:33:09,612 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Stage: Environment setup. >2014-09-09 17:33:09,697 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Stage: Environment packages setup >2014-09-09 17:33:09,738 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Stage: Environment packages setup. >2014-09-09 17:33:22,424 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum Status: Downloading Packages >2014-09-09 17:33:22,438 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Status: Downloading Packages. >2014-09-09 17:33:22,887 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum Download/Verify: 1:dmidecode-2.12-5.el6_5.x86_64 >2014-09-09 17:33:22,903 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Download/Verify: 1:dmidecode-2.12-5.el6_5.x86_64. >2014-09-09 17:33:22,904 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum Download/Verify: iproute-2.6.32-32.el6_5.x86_64 >2014-09-09 17:33:22,918 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Download/Verify: iproute-2.6.32-32.el6_5.x86_64. >2014-09-09 17:33:22,918 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum Status: Check Package Signatures >2014-09-09 17:33:22,929 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Status: Check Package Signatures. >2014-09-09 17:33:22,929 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum Status: Running Test Transaction >2014-09-09 17:33:22,943 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Status: Running Test Transaction. >2014-09-09 17:33:23,196 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum Status: Running Transaction >2014-09-09 17:33:23,213 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Status: Running Transaction. >2014-09-09 17:33:23,452 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum update: 1/4: 1:dmidecode-2.12-5.el6_5.x86_64 >2014-09-09 17:33:23,469 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum update: 1/4: 1:dmidecode-2.12-5.el6_5.x86_64. >2014-09-09 17:33:23,954 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum update: 2/4: iproute-2.6.32-32.el6_5.x86_64 >2014-09-09 17:33:23,967 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum update: 2/4: iproute-2.6.32-32.el6_5.x86_64. >2014-09-09 17:33:24,610 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum updated: 3/4: dmidecode >2014-09-09 17:33:24,624 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum updated: 3/4: dmidecode. >2014-09-09 17:33:24,942 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum updated: 4/4: iproute >2014-09-09 17:33:24,953 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum updated: 4/4: iproute. >2014-09-09 17:33:31,416 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum Verify: 1/4: iproute.x86_64 0:2.6.32-32.el6_5 - u >2014-09-09 17:33:31,431 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Verify: 1/4: iproute.x86_64 0:2.6.32-32.el6_5 - u. >2014-09-09 17:33:31,498 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum Verify: 2/4: dmidecode.x86_64 1:2.12-5.el6_5 - u >2014-09-09 17:33:31,513 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Verify: 2/4: dmidecode.x86_64 1:2.12-5.el6_5 - u. >2014-09-09 17:33:31,583 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum Verify: 3/4: dmidecode.x86_64 1:2.11-2.el6 - ud >2014-09-09 17:33:31,597 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Verify: 3/4: dmidecode.x86_64 1:2.11-2.el6 - ud. >2014-09-09 17:33:31,597 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum Verify: 4/4: iproute.x86_64 0:2.6.32-31.el6 - ud >2014-09-09 17:33:31,610 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Verify: 4/4: iproute.x86_64 0:2.6.32-31.el6 - ud. >2014-09-09 17:33:31,854 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Stage: Programs detection >2014-09-09 17:33:31,876 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Stage: Programs detection. >2014-09-09 17:33:31,876 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Stage: Environment customization >2014-09-09 17:33:31,887 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Stage: Environment customization. >2014-09-09 17:33:32,321 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Kdump supported >2014-09-09 17:33:32,333 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Kdump supported. >2014-09-09 17:33:32,374 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Logs at host located at: '/tmp/ovirt-host-deploy-20140909173234-j5oav3.log' >2014-09-09 17:33:32,388 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Logs at host located at: '/tmp/ovirt-host-deploy-20140909173234-j5oav3.log'. >2014-09-09 17:33:32,497 INFO [org.ovirt.engine.core.bll.VdsDeploy] (VdsDeploy) Host dell-r210ii-16.rhev.lab.eng.brq.redhat.com reports unique id 4C4C4544-0034-5310-8052-B4C04F4A354A >2014-09-09 17:33:32,527 INFO [org.ovirt.engine.core.bll.VdsDeploy] (VdsDeploy) Assigning unique id 4C4C4544-0034-5310-8052-B4C04F4A354A to Host dell-r210ii-16.rhev.lab.eng.brq.redhat.com >2014-09-09 17:33:32,680 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Disabling Kdump integration >2014-09-09 17:33:32,693 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Disabling Kdump integration. >2014-09-09 17:33:32,730 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Stage: Setup validation >2014-09-09 17:33:32,746 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Stage: Setup validation. >2014-09-09 17:33:32,795 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Hardware supports virtualization >2014-09-09 17:33:32,809 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Hardware supports virtualization. >2014-09-09 17:33:33,035 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Stage: Transaction setup >2014-09-09 17:33:33,049 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Stage: Transaction setup. >2014-09-09 17:33:33,127 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Stage: Misc configuration >2014-09-09 17:33:33,141 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Stage: Misc configuration. >2014-09-09 17:33:33,141 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Stage: Package installation >2014-09-09 17:33:33,155 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Stage: Package installation. >2014-09-09 17:34:03,253 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum Downloading: rhel65-brq/filelists_db (0%) >2014-09-09 17:34:03,274 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Downloading: rhel65-brq/filelists_db (0%). >2014-09-09 17:34:13,085 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum Status: Downloading Packages >2014-09-09 17:34:13,097 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Status: Downloading Packages. >2014-09-09 17:34:13,895 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum Download/Verify: audit-2.2-4.el6_5.x86_64 >2014-09-09 17:34:13,908 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Download/Verify: audit-2.2-4.el6_5.x86_64. >2014-09-09 17:34:14,547 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum Download/Verify: audit-libs-2.2-4.el6_5.x86_64 >2014-09-09 17:34:14,562 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Download/Verify: audit-libs-2.2-4.el6_5.x86_64. >2014-09-09 17:34:15,285 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum Download/Verify: audit-libs-python-2.2-4.el6_5.x86_64 >2014-09-09 17:34:15,306 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Download/Verify: audit-libs-python-2.2-4.el6_5.x86_64. >2014-09-09 17:34:15,974 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum Download/Verify: augeas-1.0.0-5.el6_5.1.x86_64 >2014-09-09 17:34:15,989 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Download/Verify: augeas-1.0.0-5.el6_5.1.x86_64. >2014-09-09 17:34:16,878 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum Download/Verify: augeas-libs-1.0.0-5.el6_5.1.x86_64 >2014-09-09 17:34:16,890 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Download/Verify: augeas-libs-1.0.0-5.el6_5.1.x86_64. >2014-09-09 17:34:17,969 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum Download/Verify: btrfs-progs-0.20-0.2.git91d9eec.el6.x86_64 >2014-09-09 17:34:17,982 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Download/Verify: btrfs-progs-0.20-0.2.git91d9eec.el6.x86_64. >2014-09-09 17:34:18,968 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum Download/Verify: celt051-0.5.1.3-0.el6.x86_64 >2014-09-09 17:34:18,982 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Download/Verify: celt051-0.5.1.3-0.el6.x86_64. >2014-09-09 17:34:20,664 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum Download/Verify: cvs-1.11.23-16.el6.x86_64 >2014-09-09 17:34:20,676 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Download/Verify: cvs-1.11.23-16.el6.x86_64. >2014-09-09 17:34:21,488 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum Download/Verify: cyrus-sasl-md5-2.1.23-13.el6_3.1.x86_64 >2014-09-09 17:34:21,499 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Download/Verify: cyrus-sasl-md5-2.1.23-13.el6_3.1.x86_64. >2014-09-09 17:34:22,501 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum Download/Verify: device-mapper-multipath-0.4.9-72.el6_5.3.x86_64 >2014-09-09 17:34:22,513 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Download/Verify: device-mapper-multipath-0.4.9-72.el6_5.3.x86_64. >2014-09-09 17:34:23,354 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum Download/Verify: device-mapper-multipath-libs-0.4.9-72.el6_5.3.x86_64 >2014-09-09 17:34:23,365 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Download/Verify: device-mapper-multipath-libs-0.4.9-72.el6_5.3.x86_64. >2014-09-09 17:34:24,288 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum Download/Verify: dnsmasq-2.48-13.el6.x86_64 >2014-09-09 17:34:24,299 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Download/Verify: dnsmasq-2.48-13.el6.x86_64. >2014-09-09 17:34:25,078 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum Download/Verify: ebtables-2.0.9-6.el6.x86_64 >2014-09-09 17:34:25,090 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Download/Verify: ebtables-2.0.9-6.el6.x86_64. >2014-09-09 17:34:26,113 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum Download/Verify: febootstrap-supermin-helper-3.21-4.el6.x86_64 >2014-09-09 17:34:26,124 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Download/Verify: febootstrap-supermin-helper-3.21-4.el6.x86_64. >2014-09-09 17:34:26,968 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum Download/Verify: fence-agents-3.1.5-35.el6_5.4.x86_64 >2014-09-09 17:34:26,980 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Download/Verify: fence-agents-3.1.5-35.el6_5.4.x86_64. >2014-09-09 17:34:27,963 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum Download/Verify: flac-1.2.1-6.1.el6.x86_64 >2014-09-09 17:34:27,974 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Download/Verify: flac-1.2.1-6.1.el6.x86_64. >2014-09-09 17:34:28,705 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum Download/Verify: fuse-2.8.3-4.el6.x86_64 >2014-09-09 17:34:28,717 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Download/Verify: fuse-2.8.3-4.el6.x86_64. >2014-09-09 17:34:29,328 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum Download/Verify: fuse-libs-2.8.3-4.el6.x86_64 >2014-09-09 17:34:29,341 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Download/Verify: fuse-libs-2.8.3-4.el6.x86_64. >2014-09-09 17:34:30,335 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum Download/Verify: genisoimage-1.1.9-12.el6.x86_64 >2014-09-09 17:34:30,349 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Download/Verify: genisoimage-1.1.9-12.el6.x86_64. >2014-09-09 17:34:32,969 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum Download/Verify: gettext-0.17-16.el6.x86_64 >2014-09-09 17:34:32,983 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Download/Verify: gettext-0.17-16.el6.x86_64. >2014-09-09 17:34:37,184 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum Download/Verify: ghostscript-8.70-19.el6.x86_64 >2014-09-09 17:34:37,196 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Download/Verify: ghostscript-8.70-19.el6.x86_64. >2014-09-09 17:34:39,880 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum Download/Verify: ghostscript-fonts-5.50-23.1.el6.noarch >2014-09-09 17:34:39,895 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Download/Verify: ghostscript-fonts-5.50-23.1.el6.noarch. >2014-09-09 17:34:42,364 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum Download/Verify: glusterfs-3.5.2-1.el6.x86_64 >2014-09-09 17:34:42,388 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Download/Verify: glusterfs-3.5.2-1.el6.x86_64. >2014-09-09 17:34:42,953 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum Download/Verify: glusterfs-api-3.5.2-1.el6.x86_64 >2014-09-09 17:34:42,967 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Download/Verify: glusterfs-api-3.5.2-1.el6.x86_64. >2014-09-09 17:34:43,989 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum Download/Verify: glusterfs-libs-3.5.2-1.el6.x86_64 >2014-09-09 17:34:44,003 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Download/Verify: glusterfs-libs-3.5.2-1.el6.x86_64. >2014-09-09 17:34:44,278 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum Download/Verify: gnutls-2.8.5-14.el6_5.x86_64 >2014-09-09 17:34:44,296 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Download/Verify: gnutls-2.8.5-14.el6_5.x86_64. >2014-09-09 17:34:44,765 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum Download/Verify: gnutls-utils-2.8.5-14.el6_5.x86_64 >2014-09-09 17:34:44,778 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Download/Verify: gnutls-utils-2.8.5-14.el6_5.x86_64. >2014-09-09 17:34:45,812 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum Download/Verify: gpxe-roms-qemu-0.9.7-6.10.el6.noarch >2014-09-09 17:34:45,824 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Download/Verify: gpxe-roms-qemu-0.9.7-6.10.el6.noarch. >2014-09-09 17:34:46,826 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum Download/Verify: hivex-1.3.3-4.2.el6.x86_64 >2014-09-09 17:34:46,837 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Download/Verify: hivex-1.3.3-4.2.el6.x86_64. >2014-09-09 17:34:46,842 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum Download/Verify: ioprocess-0.12.0-2.el6.x86_64 >2014-09-09 17:34:46,853 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Download/Verify: ioprocess-0.12.0-2.el6.x86_64. >2014-09-09 17:34:48,078 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum Download/Verify: ipmitool-1.8.11-16.el6.x86_64 >2014-09-09 17:34:48,090 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Download/Verify: ipmitool-1.8.11-16.el6.x86_64. >2014-09-09 17:34:48,721 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum Download/Verify: keyutils-1.4-4.el6.x86_64 >2014-09-09 17:34:48,732 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Download/Verify: keyutils-1.4-4.el6.x86_64. >2014-09-09 17:34:48,900 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum Download/Verify: kpartx-0.4.9-72.el6_5.3.x86_64 >2014-09-09 17:34:48,911 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Download/Verify: kpartx-0.4.9-72.el6_5.3.x86_64. >2014-09-09 17:34:49,477 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum Download/Verify: libXfont-1.4.5-3.el6_5.x86_64 >2014-09-09 17:34:49,491 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Download/Verify: libXfont-1.4.5-3.el6_5.x86_64. >2014-09-09 17:34:50,193 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum Download/Verify: libXtst-1.2.1-2.el6.x86_64 >2014-09-09 17:34:50,205 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Download/Verify: libXtst-1.2.1-2.el6.x86_64. >2014-09-09 17:34:51,047 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum Download/Verify: libasyncns-0.8-1.1.el6.x86_64 >2014-09-09 17:34:51,063 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Download/Verify: libasyncns-0.8-1.1.el6.x86_64. >2014-09-09 17:34:51,964 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum Download/Verify: libcgroup-0.40.rc1-5.el6_5.1.x86_64 >2014-09-09 17:34:51,979 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Download/Verify: libcgroup-0.40.rc1-5.el6_5.1.x86_64. >2014-09-09 17:34:52,822 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum Download/Verify: libconfig-1.3.2-1.1.el6.x86_64 >2014-09-09 17:34:52,834 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Download/Verify: libconfig-1.3.2-1.1.el6.x86_64. >2014-09-09 17:34:53,590 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum Download/Verify: libevent-1.4.13-4.el6.x86_64 >2014-09-09 17:34:53,604 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Download/Verify: libevent-1.4.13-4.el6.x86_64. >2014-09-09 17:34:54,204 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum Download/Verify: libfontenc-1.0.5-2.el6.x86_64 >2014-09-09 17:34:54,216 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Download/Verify: libfontenc-1.0.5-2.el6.x86_64. >2014-09-09 17:34:54,977 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum Download/Verify: libgomp-4.4.7-4.el6.x86_64 >2014-09-09 17:34:54,989 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Download/Verify: libgomp-4.4.7-4.el6.x86_64. >2014-09-09 17:34:55,707 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum Download/Verify: libgssglue-0.1-11.el6.x86_64 >2014-09-09 17:34:55,720 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Download/Verify: libgssglue-0.1-11.el6.x86_64. >2014-09-09 17:34:58,689 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum Download/Verify: 1:libguestfs-1.20.11-2.el6.x86_64 >2014-09-09 17:34:58,702 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Download/Verify: 1:libguestfs-1.20.11-2.el6.x86_64. >2014-09-09 17:35:01,561 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum Download/Verify: 1:libguestfs-tools-c-1.20.11-2.el6.x86_64 >2014-09-09 17:35:01,574 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Download/Verify: 1:libguestfs-tools-c-1.20.11-2.el6.x86_64. >2014-09-09 17:35:02,308 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum Download/Verify: 2:libogg-1.1.4-2.1.el6.x86_64 >2014-09-09 17:35:02,319 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Download/Verify: 2:libogg-1.1.4-2.1.el6.x86_64. >2014-09-09 17:35:03,107 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum Download/Verify: libsemanage-python-2.0.43-4.2.el6.x86_64 >2014-09-09 17:35:03,120 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Download/Verify: libsemanage-python-2.0.43-4.2.el6.x86_64. >2014-09-09 17:35:04,045 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum Download/Verify: libsndfile-1.0.20-5.el6.x86_64 >2014-09-09 17:35:04,057 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Download/Verify: libsndfile-1.0.20-5.el6.x86_64. >2014-09-09 17:35:04,775 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum Download/Verify: libtirpc-0.2.1-6.el6_5.2.x86_64 >2014-09-09 17:35:04,786 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Download/Verify: libtirpc-0.2.1-6.el6_5.2.x86_64. >2014-09-09 17:35:07,922 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum Download/Verify: libvirt-0.10.2-29.el6_5.12.x86_64 >2014-09-09 17:35:07,934 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Download/Verify: libvirt-0.10.2-29.el6_5.12.x86_64. >2014-09-09 17:35:11,835 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum Download/Verify: libvirt-client-0.10.2-29.el6_5.12.x86_64 >2014-09-09 17:35:11,849 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Download/Verify: libvirt-client-0.10.2-29.el6_5.12.x86_64. >2014-09-09 17:35:12,619 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum Download/Verify: libvirt-lock-sanlock-0.10.2-29.el6_5.12.x86_64 >2014-09-09 17:35:12,632 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Download/Verify: libvirt-lock-sanlock-0.10.2-29.el6_5.12.x86_64. >2014-09-09 17:35:14,239 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum Download/Verify: libvirt-python-0.10.2-29.el6_5.12.x86_64 >2014-09-09 17:35:14,251 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Download/Verify: libvirt-python-0.10.2-29.el6_5.12.x86_64. >2014-09-09 17:35:15,156 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum Download/Verify: 1:libvorbis-1.2.3-4.el6_2.1.x86_64 >2014-09-09 17:35:15,183 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Download/Verify: 1:libvorbis-1.2.3-4.el6_2.1.x86_64. >2014-09-09 17:35:16,117 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum Download/Verify: lm_sensors-libs-3.1.1-17.el6.x86_64 >2014-09-09 17:35:16,129 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Download/Verify: lm_sensors-libs-3.1.1-17.el6.x86_64. >2014-09-09 17:35:16,150 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum Download/Verify: lzop-1.02-0.9.rc1.el6.x86_64 >2014-09-09 17:35:16,163 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Download/Verify: lzop-1.02-0.9.rc1.el6.x86_64. >2014-09-09 17:35:16,193 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum Download/Verify: mom-0.4.1-2.el6.noarch >2014-09-09 17:35:16,205 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Download/Verify: mom-0.4.1-2.el6.noarch. >2014-09-09 17:35:17,036 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum Download/Verify: nc-1.84-22.el6.x86_64 >2014-09-09 17:35:17,050 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Download/Verify: nc-1.84-22.el6.x86_64. >2014-09-09 17:35:18,578 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum Download/Verify: 1:net-snmp-libs-5.5-49.el6_5.2.x86_64 >2014-09-09 17:35:18,591 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Download/Verify: 1:net-snmp-libs-5.5-49.el6_5.2.x86_64. >2014-09-09 17:35:19,529 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum Download/Verify: 1:net-snmp-utils-5.5-49.el6_5.2.x86_64 >2014-09-09 17:35:19,541 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Download/Verify: 1:net-snmp-utils-5.5-49.el6_5.2.x86_64. >2014-09-09 17:35:20,259 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum Download/Verify: netcf-libs-0.1.9-4.el6_5.2.x86_64 >2014-09-09 17:35:20,270 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Download/Verify: netcf-libs-0.1.9-4.el6_5.2.x86_64. >2014-09-09 17:35:22,459 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum Download/Verify: netpbm-10.47.05-11.el6.x86_64 >2014-09-09 17:35:22,470 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Download/Verify: netpbm-10.47.05-11.el6.x86_64. >2014-09-09 17:35:25,967 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum Download/Verify: netpbm-progs-10.47.05-11.el6.x86_64 >2014-09-09 17:35:25,979 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Download/Verify: netpbm-progs-10.47.05-11.el6.x86_64. >2014-09-09 17:35:27,133 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum Download/Verify: 1:nfs-utils-1.2.3-39.el6_5.3.x86_64 >2014-09-09 17:35:27,150 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Download/Verify: 1:nfs-utils-1.2.3-39.el6_5.3.x86_64. >2014-09-09 17:35:27,869 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum Download/Verify: nfs-utils-lib-1.1.5-6.el6_5.x86_64 >2014-09-09 17:35:27,882 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Download/Verify: nfs-utils-lib-1.1.5-6.el6_5.x86_64. >2014-09-09 17:35:28,652 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum Download/Verify: numad-0.5-10.20140620git.el6_5.x86_64 >2014-09-09 17:35:28,666 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Download/Verify: numad-0.5-10.20140620git.el6_5.x86_64. >2014-09-09 17:35:29,559 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum Download/Verify: perl-Net-Telnet-3.03-11.el6.noarch >2014-09-09 17:35:29,571 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Download/Verify: perl-Net-Telnet-3.03-11.el6.noarch. >2014-09-09 17:35:30,292 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum Download/Verify: perl-hivex-1.3.3-4.2.el6.x86_64 >2014-09-09 17:35:30,307 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Download/Verify: perl-hivex-1.3.3-4.2.el6.x86_64. >2014-09-09 17:35:31,124 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum Download/Verify: pexpect-2.3-6.el6.noarch >2014-09-09 17:35:31,138 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Download/Verify: pexpect-2.3-6.el6.noarch. >2014-09-09 17:35:32,397 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum Download/Verify: policycoreutils-python-2.0.83-19.39.el6.x86_64 >2014-09-09 17:35:32,413 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Download/Verify: policycoreutils-python-2.0.83-19.39.el6.x86_64. >2014-09-09 17:35:33,725 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum Download/Verify: pulseaudio-libs-0.9.21-14.el6_3.x86_64 >2014-09-09 17:35:33,736 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Download/Verify: pulseaudio-libs-0.9.21-14.el6_3.x86_64. >2014-09-09 17:35:34,679 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum Download/Verify: pyparted-3.4-4.el6.x86_64 >2014-09-09 17:35:34,692 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Download/Verify: pyparted-3.4-4.el6.x86_64. >2014-09-09 17:35:34,946 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum Download/Verify: python-argparse-1.2.1-2.1.el6.noarch >2014-09-09 17:35:34,960 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Download/Verify: python-argparse-1.2.1-2.1.el6.noarch. >2014-09-09 17:35:34,963 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum Download/Verify: python-cpopen-1.3-4.el6_5.x86_64 >2014-09-09 17:35:34,977 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Download/Verify: python-cpopen-1.3-4.el6_5.x86_64. >2014-09-09 17:35:35,507 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum Download/Verify: python-decorator-3.0.1-3.1.el6.noarch >2014-09-09 17:35:35,518 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Download/Verify: python-decorator-3.0.1-3.1.el6.noarch. >2014-09-09 17:35:35,787 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum Download/Verify: python-inotify-0.9.1-1.el6ev.noarch >2014-09-09 17:35:35,799 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Download/Verify: python-inotify-0.9.1-1.el6ev.noarch. >2014-09-09 17:35:35,800 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum Download/Verify: python-ioprocess-0.12.0-2.el6.noarch >2014-09-09 17:35:35,811 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Download/Verify: python-ioprocess-0.12.0-2.el6.noarch. >2014-09-09 17:35:37,556 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum Download/Verify: python-netaddr-0.7.5-4.el6.noarch >2014-09-09 17:35:37,567 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Download/Verify: python-netaddr-0.7.5-4.el6.noarch. >2014-09-09 17:35:37,720 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum Download/Verify: python-ordereddict-1.1-3.el6ev.noarch >2014-09-09 17:35:37,732 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Download/Verify: python-ordereddict-1.1-3.el6ev.noarch. >2014-09-09 17:35:37,820 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum Download/Verify: python-pthreading-0.1.3-3.el6ev.noarch >2014-09-09 17:35:37,831 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Download/Verify: python-pthreading-0.1.3-3.el6ev.noarch. >2014-09-09 17:35:38,968 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum Download/Verify: python-suds-0.4.1-3.el6.noarch >2014-09-09 17:35:38,982 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Download/Verify: python-suds-0.4.1-3.el6.noarch. >2014-09-09 17:35:39,929 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum Download/Verify: 2:qemu-img-rhev-0.12.1.2-2.415.el6_5.14.x86_64 >2014-09-09 17:35:39,942 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Download/Verify: 2:qemu-img-rhev-0.12.1.2-2.415.el6_5.14.x86_64. >2014-09-09 17:35:40,570 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum Download/Verify: 2:qemu-kvm-rhev-0.12.1.2-2.415.el6_5.14.x86_64 >2014-09-09 17:35:40,584 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Download/Verify: 2:qemu-kvm-rhev-0.12.1.2-2.415.el6_5.14.x86_64. >2014-09-09 17:35:40,928 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum Download/Verify: 2:qemu-kvm-rhev-tools-0.12.1.2-2.415.el6_5.14.x86_64 >2014-09-09 17:35:40,939 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Download/Verify: 2:qemu-kvm-rhev-tools-0.12.1.2-2.415.el6_5.14.x86_64. >2014-09-09 17:35:41,611 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum Download/Verify: radvd-1.6-1.el6.x86_64 >2014-09-09 17:35:41,623 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Download/Verify: radvd-1.6-1.el6.x86_64. >2014-09-09 17:35:42,410 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum Download/Verify: rpcbind-0.2.0-11.el6.x86_64 >2014-09-09 17:35:42,425 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Download/Verify: rpcbind-0.2.0-11.el6.x86_64. >2014-09-09 17:35:43,190 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum Download/Verify: sanlock-2.8-1.el6.x86_64 >2014-09-09 17:35:43,205 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Download/Verify: sanlock-2.8-1.el6.x86_64. >2014-09-09 17:35:43,816 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum Download/Verify: sanlock-lib-2.8-1.el6.x86_64 >2014-09-09 17:35:43,833 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Download/Verify: sanlock-lib-2.8-1.el6.x86_64. >2014-09-09 17:35:44,569 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum Download/Verify: sanlock-python-2.8-1.el6.x86_64 >2014-09-09 17:35:44,584 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Download/Verify: sanlock-python-2.8-1.el6.x86_64. >2014-09-09 17:35:45,178 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum Download/Verify: scrub-2.2-1.el6.x86_64 >2014-09-09 17:35:45,196 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Download/Verify: scrub-2.2-1.el6.x86_64. >2014-09-09 17:35:46,030 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum Download/Verify: seabios-0.6.1.2-28.el6.x86_64 >2014-09-09 17:35:46,045 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Download/Verify: seabios-0.6.1.2-28.el6.x86_64. >2014-09-09 17:35:47,163 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum Download/Verify: setools-libs-3.3.7-4.el6.x86_64 >2014-09-09 17:35:47,178 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Download/Verify: setools-libs-3.3.7-4.el6.x86_64. >2014-09-09 17:35:48,096 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum Download/Verify: setools-libs-python-3.3.7-4.el6.x86_64 >2014-09-09 17:35:48,111 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Download/Verify: setools-libs-python-3.3.7-4.el6.x86_64. >2014-09-09 17:35:49,552 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum Download/Verify: sg3_utils-1.28-5.el6.x86_64 >2014-09-09 17:35:49,565 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Download/Verify: sg3_utils-1.28-5.el6.x86_64. >2014-09-09 17:35:50,080 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum Download/Verify: sgabios-bin-0-0.3.20110621svn.el6.noarch >2014-09-09 17:35:50,094 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Download/Verify: sgabios-bin-0-0.3.20110621svn.el6.noarch. >2014-09-09 17:35:50,958 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum Download/Verify: spice-server-0.12.4-6.el6_5.2.x86_64 >2014-09-09 17:35:50,973 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Download/Verify: spice-server-0.12.4-6.el6_5.2.x86_64. >2014-09-09 17:35:51,635 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum Download/Verify: 1:telnet-0.17-47.el6_3.1.x86_64 >2014-09-09 17:35:51,651 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Download/Verify: 1:telnet-0.17-47.el6_3.1.x86_64. >2014-09-09 17:35:52,347 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum Download/Verify: tree-1.5.3-2.el6.x86_64 >2014-09-09 17:35:52,358 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Download/Verify: tree-1.5.3-2.el6.x86_64. >2014-09-09 17:35:55,039 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum Download/Verify: urw-fonts-2.4-10.el6.noarch >2014-09-09 17:35:55,055 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Download/Verify: urw-fonts-2.4-10.el6.noarch. >2014-09-09 17:35:55,685 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum Download/Verify: usbredir-0.5.1-1.el6.x86_64 >2014-09-09 17:35:55,699 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Download/Verify: usbredir-0.5.1-1.el6.x86_64. >2014-09-09 17:35:55,716 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum Download/Verify: vdsm-4.16.3-2.el6.x86_64 >2014-09-09 17:35:55,731 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Download/Verify: vdsm-4.16.3-2.el6.x86_64. >2014-09-09 17:35:55,731 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum Download/Verify: vdsm-cli-4.16.3-2.el6.noarch >2014-09-09 17:35:55,745 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Download/Verify: vdsm-cli-4.16.3-2.el6.noarch. >2014-09-09 17:35:55,745 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum Download/Verify: vdsm-jsonrpc-4.16.3-2.el6.noarch >2014-09-09 17:35:55,757 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Download/Verify: vdsm-jsonrpc-4.16.3-2.el6.noarch. >2014-09-09 17:35:55,757 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum Download/Verify: vdsm-python-4.16.3-2.el6.x86_64 >2014-09-09 17:35:55,769 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Download/Verify: vdsm-python-4.16.3-2.el6.x86_64. >2014-09-09 17:35:55,769 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum Download/Verify: vdsm-python-zombiereaper-4.16.3-2.el6.noarch >2014-09-09 17:35:55,781 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Download/Verify: vdsm-python-zombiereaper-4.16.3-2.el6.noarch. >2014-09-09 17:35:55,782 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum Download/Verify: vdsm-xmlrpc-4.16.3-2.el6.noarch >2014-09-09 17:35:55,794 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Download/Verify: vdsm-xmlrpc-4.16.3-2.el6.noarch. >2014-09-09 17:35:55,795 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum Download/Verify: vdsm-yajsonrpc-4.16.3-2.el6.noarch >2014-09-09 17:35:55,806 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Download/Verify: vdsm-yajsonrpc-4.16.3-2.el6.noarch. >2014-09-09 17:35:56,452 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum Download/Verify: vgabios-0.6b-3.7.el6.noarch >2014-09-09 17:35:56,472 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Download/Verify: vgabios-0.6b-3.7.el6.noarch. >2014-09-09 17:35:57,279 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum Download/Verify: 1:xorg-x11-font-utils-7.2-11.el6.x86_64 >2014-09-09 17:35:57,293 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Download/Verify: 1:xorg-x11-font-utils-7.2-11.el6.x86_64. >2014-09-09 17:35:57,972 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum Download/Verify: yajl-1.0.7-3.el6.x86_64 >2014-09-09 17:35:57,987 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Download/Verify: yajl-1.0.7-3.el6.x86_64. >2014-09-09 17:35:57,988 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum Status: Check Package Signatures >2014-09-09 17:35:58,002 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Status: Check Package Signatures. >2014-09-09 17:35:58,003 WARN [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Confirm use of GPG Key userid=Red Hat, Inc. (release key 2) <security@redhat.com> hexkeyid=FD431D51 >2014-09-09 17:35:58,017 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Host rack installation in progress . Confirm use of GPG Key userid=Red Hat, Inc. (release key 2) <security@redhat.com> hexkeyid=FD431D51. >2014-09-09 17:35:58,172 WARN [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Confirm use of GPG Key userid=Red Hat, Inc. (auxiliary key) <security@redhat.com> hexkeyid=2FA658E0 >2014-09-09 17:35:58,186 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Host rack installation in progress . Confirm use of GPG Key userid=Red Hat, Inc. (auxiliary key) <security@redhat.com> hexkeyid=2FA658E0. >2014-09-09 17:35:58,309 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum Status: Running Test Transaction >2014-09-09 17:35:58,323 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Status: Running Test Transaction. >2014-09-09 17:35:58,971 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum Status: Running Transaction >2014-09-09 17:35:58,987 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Status: Running Transaction. >2014-09-09 17:35:59,423 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum update: 1/113: audit-libs-2.2-4.el6_5.x86_64 >2014-09-09 17:35:59,438 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum update: 1/113: audit-libs-2.2-4.el6_5.x86_64. >2014-09-09 17:35:59,834 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum install: 2/113: yajl-1.0.7-3.el6.x86_64 >2014-09-09 17:35:59,853 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum install: 2/113: yajl-1.0.7-3.el6.x86_64. >2014-09-09 17:36:00,240 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum update: 3/113: gnutls-2.8.5-14.el6_5.x86_64 >2014-09-09 17:36:00,252 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum update: 3/113: gnutls-2.8.5-14.el6_5.x86_64. >2014-09-09 17:36:00,857 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum install: 4/113: libgssglue-0.1-11.el6.x86_64 >2014-09-09 17:36:00,870 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum install: 4/113: libgssglue-0.1-11.el6.x86_64. >2014-09-09 17:36:01,332 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum install: 5/113: glusterfs-libs-3.5.2-1.el6.x86_64 >2014-09-09 17:36:01,346 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum install: 5/113: glusterfs-libs-3.5.2-1.el6.x86_64. >2014-09-09 17:36:01,771 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum install: 6/113: 2:libogg-1.1.4-2.1.el6.x86_64 >2014-09-09 17:36:01,789 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum install: 6/113: 2:libogg-1.1.4-2.1.el6.x86_64. >2014-09-09 17:36:02,191 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum install: 7/113: augeas-libs-1.0.0-5.el6_5.1.x86_64 >2014-09-09 17:36:02,205 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum install: 7/113: augeas-libs-1.0.0-5.el6_5.1.x86_64. >2014-09-09 17:36:02,890 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum install: 8/113: sanlock-lib-2.8-1.el6.x86_64 >2014-09-09 17:36:02,912 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum install: 8/113: sanlock-lib-2.8-1.el6.x86_64. >2014-09-09 17:36:03,557 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum install: 9/113: libcgroup-0.40.rc1-5.el6_5.1.x86_64 >2014-09-09 17:36:03,572 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum install: 9/113: libcgroup-0.40.rc1-5.el6_5.1.x86_64. >2014-09-09 17:36:04,166 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum install: 10/113: python-cpopen-1.3-4.el6_5.x86_64 >2014-09-09 17:36:04,181 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum install: 10/113: python-cpopen-1.3-4.el6_5.x86_64. >2014-09-09 17:36:05,102 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum install: 11/113: sanlock-2.8-1.el6.x86_64 >2014-09-09 17:36:05,117 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum install: 11/113: sanlock-2.8-1.el6.x86_64. >2014-09-09 17:36:05,515 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum install: 12/113: libtirpc-0.2.1-6.el6_5.2.x86_64 >2014-09-09 17:36:05,531 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum install: 12/113: libtirpc-0.2.1-6.el6_5.2.x86_64. >2014-09-09 17:36:05,932 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum install: 13/113: perl-hivex-1.3.3-4.2.el6.x86_64 >2014-09-09 17:36:05,946 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum install: 13/113: perl-hivex-1.3.3-4.2.el6.x86_64. >2014-09-09 17:36:06,448 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum install: 14/113: hivex-1.3.3-4.2.el6.x86_64 >2014-09-09 17:36:06,463 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum install: 14/113: hivex-1.3.3-4.2.el6.x86_64. >2014-09-09 17:36:06,938 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum install: 15/113: lzop-1.02-0.9.rc1.el6.x86_64 >2014-09-09 17:36:06,951 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum install: 15/113: lzop-1.02-0.9.rc1.el6.x86_64. >2014-09-09 17:36:07,331 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum install: 16/113: libfontenc-1.0.5-2.el6.x86_64 >2014-09-09 17:36:07,346 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum install: 16/113: libfontenc-1.0.5-2.el6.x86_64. >2014-09-09 17:36:07,748 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum install: 17/113: fuse-libs-2.8.3-4.el6.x86_64 >2014-09-09 17:36:07,759 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum install: 17/113: fuse-libs-2.8.3-4.el6.x86_64. >2014-09-09 17:36:08,156 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum install: 18/113: usbredir-0.5.1-1.el6.x86_64 >2014-09-09 17:36:08,168 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum install: 18/113: usbredir-0.5.1-1.el6.x86_64. >2014-09-09 17:36:08,573 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum install: 19/113: vdsm-python-zombiereaper-4.16.3-2.el6.noarch >2014-09-09 17:36:08,587 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum install: 19/113: vdsm-python-zombiereaper-4.16.3-2.el6.noarch. >2014-09-09 17:36:08,848 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum install: 20/113: libXfont-1.4.5-3.el6_5.x86_64 >2014-09-09 17:36:08,861 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum install: 20/113: libXfont-1.4.5-3.el6_5.x86_64. >2014-09-09 17:36:09,315 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum install: 21/113: 1:xorg-x11-font-utils-7.2-11.el6.x86_64 >2014-09-09 17:36:09,326 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum install: 21/113: 1:xorg-x11-font-utils-7.2-11.el6.x86_64. >2014-09-09 17:36:09,726 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum install: 22/113: ghostscript-fonts-5.50-23.1.el6.noarch >2014-09-09 17:36:09,740 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum install: 22/113: ghostscript-fonts-5.50-23.1.el6.noarch. >2014-09-09 17:36:12,619 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum install: 23/113: rpcbind-0.2.0-11.el6.x86_64 >2014-09-09 17:36:12,635 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum install: 23/113: rpcbind-0.2.0-11.el6.x86_64. >2014-09-09 17:36:13,032 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum install: 24/113: numad-0.5-10.20140620git.el6_5.x86_64 >2014-09-09 17:36:13,043 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum install: 24/113: numad-0.5-10.20140620git.el6_5.x86_64. >2014-09-09 17:36:13,473 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum install: 25/113: sanlock-python-2.8-1.el6.x86_64 >2014-09-09 17:36:13,485 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum install: 25/113: sanlock-python-2.8-1.el6.x86_64. >2014-09-09 17:36:13,832 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum install: 26/113: augeas-1.0.0-5.el6_5.1.x86_64 >2014-09-09 17:36:13,849 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum install: 26/113: augeas-1.0.0-5.el6_5.1.x86_64. >2014-09-09 17:36:14,298 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum install: 27/113: netcf-libs-0.1.9-4.el6_5.2.x86_64 >2014-09-09 17:36:14,311 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum install: 27/113: netcf-libs-0.1.9-4.el6_5.2.x86_64. >2014-09-09 17:36:14,824 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum install: 28/113: celt051-0.5.1.3-0.el6.x86_64 >2014-09-09 17:36:14,835 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum install: 28/113: celt051-0.5.1.3-0.el6.x86_64. >2014-09-09 17:36:15,265 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum install: 29/113: spice-server-0.12.4-6.el6_5.2.x86_64 >2014-09-09 17:36:15,277 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum install: 29/113: spice-server-0.12.4-6.el6_5.2.x86_64. >2014-09-09 17:36:15,762 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum install: 30/113: flac-1.2.1-6.1.el6.x86_64 >2014-09-09 17:36:15,775 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum install: 30/113: flac-1.2.1-6.1.el6.x86_64. >2014-09-09 17:36:16,198 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum install: 31/113: 1:libvorbis-1.2.3-4.el6_2.1.x86_64 >2014-09-09 17:36:16,209 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum install: 31/113: 1:libvorbis-1.2.3-4.el6_2.1.x86_64. >2014-09-09 17:36:16,631 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum install: 32/113: libsndfile-1.0.20-5.el6.x86_64 >2014-09-09 17:36:16,643 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum install: 32/113: libsndfile-1.0.20-5.el6.x86_64. >2014-09-09 17:36:17,075 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum install: 33/113: glusterfs-3.5.2-1.el6.x86_64 >2014-09-09 17:36:17,090 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum install: 33/113: glusterfs-3.5.2-1.el6.x86_64. >2014-09-09 17:36:17,716 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum install: 34/113: glusterfs-api-3.5.2-1.el6.x86_64 >2014-09-09 17:36:17,729 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum install: 34/113: glusterfs-api-3.5.2-1.el6.x86_64. >2014-09-09 17:36:18,157 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum install: 35/113: 2:qemu-img-rhev-0.12.1.2-2.415.el6_5.14.x86_64 >2014-09-09 17:36:18,170 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum install: 35/113: 2:qemu-img-rhev-0.12.1.2-2.415.el6_5.14.x86_64. >2014-09-09 17:36:18,545 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum install: 36/113: gnutls-utils-2.8.5-14.el6_5.x86_64 >2014-09-09 17:36:18,555 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum install: 36/113: gnutls-utils-2.8.5-14.el6_5.x86_64. >2014-09-09 17:36:18,909 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum install: 37/113: ioprocess-0.12.0-2.el6.x86_64 >2014-09-09 17:36:18,922 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum install: 37/113: ioprocess-0.12.0-2.el6.x86_64. >2014-09-09 17:36:19,248 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum install: 38/113: python-ioprocess-0.12.0-2.el6.noarch >2014-09-09 17:36:19,260 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum install: 38/113: python-ioprocess-0.12.0-2.el6.noarch. >2014-09-09 17:36:19,574 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum install: 39/113: vdsm-python-4.16.3-2.el6.x86_64 >2014-09-09 17:36:19,589 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum install: 39/113: vdsm-python-4.16.3-2.el6.x86_64. >2014-09-09 17:36:19,989 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum install: 40/113: vdsm-xmlrpc-4.16.3-2.el6.noarch >2014-09-09 17:36:20,009 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum install: 40/113: vdsm-xmlrpc-4.16.3-2.el6.noarch. >2014-09-09 17:36:20,301 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum install: 41/113: vdsm-cli-4.16.3-2.el6.noarch >2014-09-09 17:36:20,315 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum install: 41/113: vdsm-cli-4.16.3-2.el6.noarch. >2014-09-09 17:36:20,643 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum install: 42/113: audit-libs-python-2.2-4.el6_5.x86_64 >2014-09-09 17:36:20,658 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum install: 42/113: audit-libs-python-2.2-4.el6_5.x86_64. >2014-09-09 17:36:20,989 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum install: 43/113: lm_sensors-libs-3.1.1-17.el6.x86_64 >2014-09-09 17:36:21,008 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum install: 43/113: lm_sensors-libs-3.1.1-17.el6.x86_64. >2014-09-09 17:36:21,419 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum install: 44/113: 1:net-snmp-libs-5.5-49.el6_5.2.x86_64 >2014-09-09 17:36:21,432 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum install: 44/113: 1:net-snmp-libs-5.5-49.el6_5.2.x86_64. >2014-09-09 17:36:22,050 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum install: 45/113: 1:net-snmp-utils-5.5-49.el6_5.2.x86_64 >2014-09-09 17:36:22,064 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum install: 45/113: 1:net-snmp-utils-5.5-49.el6_5.2.x86_64. >2014-09-09 17:36:22,476 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum install: 46/113: cyrus-sasl-md5-2.1.23-13.el6_3.1.x86_64 >2014-09-09 17:36:22,488 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum install: 46/113: cyrus-sasl-md5-2.1.23-13.el6_3.1.x86_64. >2014-09-09 17:36:22,847 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum install: 47/113: ipmitool-1.8.11-16.el6.x86_64 >2014-09-09 17:36:22,859 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum install: 47/113: ipmitool-1.8.11-16.el6.x86_64. >2014-09-09 17:36:23,340 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum install: 48/113: libgomp-4.4.7-4.el6.x86_64 >2014-09-09 17:36:23,355 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum install: 48/113: libgomp-4.4.7-4.el6.x86_64. >2014-09-09 17:36:23,782 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum install: 49/113: python-netaddr-0.7.5-4.el6.noarch >2014-09-09 17:36:23,795 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum install: 49/113: python-netaddr-0.7.5-4.el6.noarch. >2014-09-09 17:36:24,281 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum install: 50/113: dnsmasq-2.48-13.el6.x86_64 >2014-09-09 17:36:24,295 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum install: 50/113: dnsmasq-2.48-13.el6.x86_64. >2014-09-09 17:36:24,665 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum install: 51/113: python-ordereddict-1.1-3.el6ev.noarch >2014-09-09 17:36:24,679 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum install: 51/113: python-ordereddict-1.1-3.el6ev.noarch. >2014-09-09 17:36:24,974 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum install: 52/113: sg3_utils-1.28-5.el6.x86_64 >2014-09-09 17:36:24,988 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum install: 52/113: sg3_utils-1.28-5.el6.x86_64. >2014-09-09 17:36:25,415 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum install: 53/113: python-argparse-1.2.1-2.1.el6.noarch >2014-09-09 17:36:25,436 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum install: 53/113: python-argparse-1.2.1-2.1.el6.noarch. >2014-09-09 17:36:25,714 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum install: 54/113: cvs-1.11.23-16.el6.x86_64 >2014-09-09 17:36:25,731 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum install: 54/113: cvs-1.11.23-16.el6.x86_64. >2014-09-09 17:36:26,234 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum install: 55/113: gettext-0.17-16.el6.x86_64 >2014-09-09 17:36:26,246 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum install: 55/113: gettext-0.17-16.el6.x86_64. >2014-09-09 17:36:27,006 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum install: 56/113: perl-Net-Telnet-3.03-11.el6.noarch >2014-09-09 17:36:27,033 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum install: 56/113: perl-Net-Telnet-3.03-11.el6.noarch. >2014-09-09 17:36:27,356 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum install: 57/113: libXtst-1.2.1-2.el6.x86_64 >2014-09-09 17:36:27,370 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum install: 57/113: libXtst-1.2.1-2.el6.x86_64. >2014-09-09 17:36:27,714 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum install: 58/113: seabios-0.6.1.2-28.el6.x86_64 >2014-09-09 17:36:27,729 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum install: 58/113: seabios-0.6.1.2-28.el6.x86_64. >2014-09-09 17:36:27,998 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum install: 59/113: python-suds-0.4.1-3.el6.noarch >2014-09-09 17:36:28,012 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum install: 59/113: python-suds-0.4.1-3.el6.noarch. >2014-09-09 17:36:28,462 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum install: 60/113: 1:telnet-0.17-47.el6_3.1.x86_64 >2014-09-09 17:36:28,478 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum install: 60/113: 1:telnet-0.17-47.el6_3.1.x86_64. >2014-09-09 17:36:28,831 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum install: 61/113: urw-fonts-2.4-10.el6.noarch >2014-09-09 17:36:28,845 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum install: 61/113: urw-fonts-2.4-10.el6.noarch. >2014-09-09 17:36:31,678 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum install: 62/113: ghostscript-8.70-19.el6.x86_64 >2014-09-09 17:36:31,693 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum install: 62/113: ghostscript-8.70-19.el6.x86_64. >2014-09-09 17:36:32,906 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum install: 63/113: libevent-1.4.13-4.el6.x86_64 >2014-09-09 17:36:32,920 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum install: 63/113: libevent-1.4.13-4.el6.x86_64. >2014-09-09 17:36:33,348 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum install: 64/113: python-inotify-0.9.1-1.el6ev.noarch >2014-09-09 17:36:33,363 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum install: 64/113: python-inotify-0.9.1-1.el6ev.noarch. >2014-09-09 17:36:33,706 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum install: 65/113: ebtables-2.0.9-6.el6.x86_64 >2014-09-09 17:36:33,721 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum install: 65/113: ebtables-2.0.9-6.el6.x86_64. >2014-09-09 17:36:34,165 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum install: 66/113: pexpect-2.3-6.el6.noarch >2014-09-09 17:36:34,185 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum install: 66/113: pexpect-2.3-6.el6.noarch. >2014-09-09 17:36:34,574 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum install: 67/113: keyutils-1.4-4.el6.x86_64 >2014-09-09 17:36:34,584 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum install: 67/113: keyutils-1.4-4.el6.x86_64. >2014-09-09 17:36:34,906 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum install: 68/113: nfs-utils-lib-1.1.5-6.el6_5.x86_64 >2014-09-09 17:36:34,917 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum install: 68/113: nfs-utils-lib-1.1.5-6.el6_5.x86_64. >2014-09-09 17:36:35,856 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum install: 69/113: 1:nfs-utils-1.2.3-39.el6_5.3.x86_64 >2014-09-09 17:36:35,869 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum install: 69/113: 1:nfs-utils-1.2.3-39.el6_5.3.x86_64. >2014-09-09 17:36:36,662 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum install: 70/113: btrfs-progs-0.20-0.2.git91d9eec.el6.x86_64 >2014-09-09 17:36:36,678 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum install: 70/113: btrfs-progs-0.20-0.2.git91d9eec.el6.x86_64. >2014-09-09 17:36:37,120 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum install: 71/113: genisoimage-1.1.9-12.el6.x86_64 >2014-09-09 17:36:37,136 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum install: 71/113: genisoimage-1.1.9-12.el6.x86_64. >2014-09-09 17:36:37,567 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum install: 72/113: scrub-2.2-1.el6.x86_64 >2014-09-09 17:36:37,582 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum install: 72/113: scrub-2.2-1.el6.x86_64. >2014-09-09 17:36:37,889 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum install: 73/113: setools-libs-3.3.7-4.el6.x86_64 >2014-09-09 17:36:37,903 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum install: 73/113: setools-libs-3.3.7-4.el6.x86_64. >2014-09-09 17:36:38,372 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum install: 74/113: setools-libs-python-3.3.7-4.el6.x86_64 >2014-09-09 17:36:38,384 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum install: 74/113: setools-libs-python-3.3.7-4.el6.x86_64. >2014-09-09 17:36:39,116 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum install: 75/113: radvd-1.6-1.el6.x86_64 >2014-09-09 17:36:39,128 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum install: 75/113: radvd-1.6-1.el6.x86_64. >2014-09-09 17:36:39,539 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum install: 76/113: libasyncns-0.8-1.1.el6.x86_64 >2014-09-09 17:36:39,555 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum install: 76/113: libasyncns-0.8-1.1.el6.x86_64. >2014-09-09 17:36:39,931 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum install: 77/113: pulseaudio-libs-0.9.21-14.el6_3.x86_64 >2014-09-09 17:36:39,948 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum install: 77/113: pulseaudio-libs-0.9.21-14.el6_3.x86_64. >2014-09-09 17:36:40,467 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum update: 78/113: kpartx-0.4.9-72.el6_5.3.x86_64 >2014-09-09 17:36:40,481 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum update: 78/113: kpartx-0.4.9-72.el6_5.3.x86_64. >2014-09-09 17:36:40,837 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum install: 79/113: febootstrap-supermin-helper-3.21-4.el6.x86_64 >2014-09-09 17:36:40,849 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum install: 79/113: febootstrap-supermin-helper-3.21-4.el6.x86_64. >2014-09-09 17:36:41,222 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum install: 80/113: python-decorator-3.0.1-3.1.el6.noarch >2014-09-09 17:36:41,238 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum install: 80/113: python-decorator-3.0.1-3.1.el6.noarch. >2014-09-09 17:36:41,701 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum install: 81/113: pyparted-3.4-4.el6.x86_64 >2014-09-09 17:36:41,718 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum install: 81/113: pyparted-3.4-4.el6.x86_64. >2014-09-09 17:36:42,109 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum install: 82/113: libsemanage-python-2.0.43-4.2.el6.x86_64 >2014-09-09 17:36:42,123 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum install: 82/113: libsemanage-python-2.0.43-4.2.el6.x86_64. >2014-09-09 17:36:42,460 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum install: 83/113: policycoreutils-python-2.0.83-19.39.el6.x86_64 >2014-09-09 17:36:42,474 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum install: 83/113: policycoreutils-python-2.0.83-19.39.el6.x86_64. >2014-09-09 17:37:10,753 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum install: 84/113: fuse-2.8.3-4.el6.x86_64 >2014-09-09 17:37:10,768 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum install: 84/113: fuse-2.8.3-4.el6.x86_64. >2014-09-09 17:37:11,163 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum install: 85/113: vdsm-yajsonrpc-4.16.3-2.el6.noarch >2014-09-09 17:37:11,179 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum install: 85/113: vdsm-yajsonrpc-4.16.3-2.el6.noarch. >2014-09-09 17:37:11,504 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum install: 86/113: vdsm-jsonrpc-4.16.3-2.el6.noarch >2014-09-09 17:37:11,516 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum install: 86/113: vdsm-jsonrpc-4.16.3-2.el6.noarch. >2014-09-09 17:37:12,130 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum install: 87/113: python-pthreading-0.1.3-3.el6ev.noarch >2014-09-09 17:37:12,143 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum install: 87/113: python-pthreading-0.1.3-3.el6ev.noarch. >2014-09-09 17:37:12,523 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum install: 88/113: nc-1.84-22.el6.x86_64 >2014-09-09 17:37:12,538 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum install: 88/113: nc-1.84-22.el6.x86_64. >2014-09-09 17:37:12,917 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum install: 89/113: libvirt-client-0.10.2-29.el6_5.12.x86_64 >2014-09-09 17:37:12,932 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum install: 89/113: libvirt-client-0.10.2-29.el6_5.12.x86_64. >2014-09-09 17:37:14,429 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum install: 90/113: libvirt-0.10.2-29.el6_5.12.x86_64 >2014-09-09 17:37:14,440 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum install: 90/113: libvirt-0.10.2-29.el6_5.12.x86_64. >2014-09-09 17:37:15,184 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum install: 91/113: libvirt-python-0.10.2-29.el6_5.12.x86_64 >2014-09-09 17:37:15,197 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum install: 91/113: libvirt-python-0.10.2-29.el6_5.12.x86_64. >2014-09-09 17:37:15,631 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum install: 92/113: mom-0.4.1-2.el6.noarch >2014-09-09 17:37:15,645 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum install: 92/113: mom-0.4.1-2.el6.noarch. >2014-09-09 17:37:16,090 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum install: 93/113: libvirt-lock-sanlock-0.10.2-29.el6_5.12.x86_64 >2014-09-09 17:37:16,104 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum install: 93/113: libvirt-lock-sanlock-0.10.2-29.el6_5.12.x86_64. >2014-09-09 17:37:16,540 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum install: 94/113: fence-agents-3.1.5-35.el6_5.4.x86_64 >2014-09-09 17:37:16,551 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum install: 94/113: fence-agents-3.1.5-35.el6_5.4.x86_64. >2014-09-09 17:37:17,414 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum install: 95/113: netpbm-10.47.05-11.el6.x86_64 >2014-09-09 17:37:17,432 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum install: 95/113: netpbm-10.47.05-11.el6.x86_64. >2014-09-09 17:37:18,123 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum install: 96/113: netpbm-progs-10.47.05-11.el6.x86_64 >2014-09-09 17:37:18,137 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum install: 96/113: netpbm-progs-10.47.05-11.el6.x86_64. >2014-09-09 17:37:18,954 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum install: 97/113: gpxe-roms-qemu-0.9.7-6.10.el6.noarch >2014-09-09 17:37:18,968 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum install: 97/113: gpxe-roms-qemu-0.9.7-6.10.el6.noarch. >2014-09-09 17:37:19,320 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum install: 98/113: sgabios-bin-0-0.3.20110621svn.el6.noarch >2014-09-09 17:37:19,337 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum install: 98/113: sgabios-bin-0-0.3.20110621svn.el6.noarch. >2014-09-09 17:37:19,624 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum install: 99/113: tree-1.5.3-2.el6.x86_64 >2014-09-09 17:37:19,639 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum install: 99/113: tree-1.5.3-2.el6.x86_64. >2014-09-09 17:37:19,962 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum install: 100/113: vgabios-0.6b-3.7.el6.noarch >2014-09-09 17:37:19,977 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum install: 100/113: vgabios-0.6b-3.7.el6.noarch. >2014-09-09 17:37:20,319 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum install: 101/113: 2:qemu-kvm-rhev-0.12.1.2-2.415.el6_5.14.x86_64 >2014-09-09 17:37:20,333 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum install: 101/113: 2:qemu-kvm-rhev-0.12.1.2-2.415.el6_5.14.x86_64. >2014-09-09 17:37:21,107 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum install: 102/113: 1:libguestfs-1.20.11-2.el6.x86_64 >2014-09-09 17:37:21,122 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum install: 102/113: 1:libguestfs-1.20.11-2.el6.x86_64. >2014-09-09 17:37:21,754 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum install: 103/113: device-mapper-multipath-libs-0.4.9-72.el6_5.3.x86_64 >2014-09-09 17:37:21,768 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum install: 103/113: device-mapper-multipath-libs-0.4.9-72.el6_5.3.x86_64. >2014-09-09 17:37:22,271 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum install: 104/113: device-mapper-multipath-0.4.9-72.el6_5.3.x86_64 >2014-09-09 17:37:22,286 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum install: 104/113: device-mapper-multipath-0.4.9-72.el6_5.3.x86_64. >2014-09-09 17:37:22,746 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum install: 105/113: libconfig-1.3.2-1.1.el6.x86_64 >2014-09-09 17:37:22,759 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum install: 105/113: libconfig-1.3.2-1.1.el6.x86_64. >2014-09-09 17:37:23,204 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum install: 106/113: 1:libguestfs-tools-c-1.20.11-2.el6.x86_64 >2014-09-09 17:37:23,216 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum install: 106/113: 1:libguestfs-tools-c-1.20.11-2.el6.x86_64. >2014-09-09 17:37:24,220 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum install: 107/113: vdsm-4.16.3-2.el6.x86_64 >2014-09-09 17:37:24,234 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum install: 107/113: vdsm-4.16.3-2.el6.x86_64. >2014-09-09 17:37:48,215 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum update: 108/113: audit-2.2-4.el6_5.x86_64 >2014-09-09 17:37:48,230 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum update: 108/113: audit-2.2-4.el6_5.x86_64. >2014-09-09 17:37:48,784 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum install: 109/113: 2:qemu-kvm-rhev-tools-0.12.1.2-2.415.el6_5.14.x86_64 >2014-09-09 17:37:48,797 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum install: 109/113: 2:qemu-kvm-rhev-tools-0.12.1.2-2.415.el6_5.14.x86_64. >2014-09-09 17:37:49,165 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum updated: 110/113: audit >2014-09-09 17:37:49,175 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum updated: 110/113: audit. >2014-09-09 17:37:49,789 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum updated: 111/113: audit-libs >2014-09-09 17:37:49,801 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum updated: 111/113: audit-libs. >2014-09-09 17:37:50,106 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum updated: 112/113: kpartx >2014-09-09 17:37:50,119 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum updated: 112/113: kpartx. >2014-09-09 17:37:50,382 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum updated: 113/113: gnutls >2014-09-09 17:37:50,401 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum updated: 113/113: gnutls. >2014-09-09 17:38:02,755 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum Verify: 1/113: libconfig.x86_64 0:1.3.2-1.1.el6 - u >2014-09-09 17:38:02,772 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Verify: 1/113: libconfig.x86_64 0:1.3.2-1.1.el6 - u. >2014-09-09 17:38:02,824 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum Verify: 2/113: device-mapper-multipath-libs.x86_64 0:0.4.9-72.el6_5.3 - u >2014-09-09 17:38:02,838 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Verify: 2/113: device-mapper-multipath-libs.x86_64 0:0.4.9-72.el6_5.3 - u. >2014-09-09 17:38:02,916 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum Verify: 3/113: vgabios.noarch 0:0.6b-3.7.el6 - u >2014-09-09 17:38:02,930 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Verify: 3/113: vgabios.noarch 0:0.6b-3.7.el6 - u. >2014-09-09 17:38:03,009 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum Verify: 4/113: tree.x86_64 0:1.5.3-2.el6 - u >2014-09-09 17:38:03,024 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Verify: 4/113: tree.x86_64 0:1.5.3-2.el6 - u. >2014-09-09 17:38:03,082 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum Verify: 5/113: sgabios-bin.noarch 0:0-0.3.20110621svn.el6 - u >2014-09-09 17:38:03,096 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Verify: 5/113: sgabios-bin.noarch 0:0-0.3.20110621svn.el6 - u. >2014-09-09 17:38:03,194 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum Verify: 6/113: qemu-img-rhev.x86_64 2:0.12.1.2-2.415.el6_5.14 - u >2014-09-09 17:38:03,207 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Verify: 6/113: qemu-img-rhev.x86_64 2:0.12.1.2-2.415.el6_5.14 - u. >2014-09-09 17:38:03,265 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum Verify: 7/113: celt051.x86_64 0:0.5.1.3-0.el6 - u >2014-09-09 17:38:03,281 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Verify: 7/113: celt051.x86_64 0:0.5.1.3-0.el6 - u. >2014-09-09 17:38:03,386 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum Verify: 8/113: sanlock.x86_64 0:2.8-1.el6 - u >2014-09-09 17:38:03,402 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Verify: 8/113: sanlock.x86_64 0:2.8-1.el6 - u. >2014-09-09 17:38:03,479 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum Verify: 9/113: qemu-kvm-rhev-tools.x86_64 2:0.12.1.2-2.415.el6_5.14 - u >2014-09-09 17:38:03,494 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Verify: 9/113: qemu-kvm-rhev-tools.x86_64 2:0.12.1.2-2.415.el6_5.14 - u. >2014-09-09 17:38:03,571 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum Verify: 10/113: hivex.x86_64 0:1.3.3-4.2.el6 - u >2014-09-09 17:38:03,586 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Verify: 10/113: hivex.x86_64 0:1.3.3-4.2.el6 - u. >2014-09-09 17:38:03,665 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum Verify: 11/113: ghostscript-fonts.noarch 0:5.50-23.1.el6 - u >2014-09-09 17:38:03,679 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Verify: 11/113: ghostscript-fonts.noarch 0:5.50-23.1.el6 - u. >2014-09-09 17:38:03,715 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum Verify: 12/113: gettext.x86_64 0:0.17-16.el6 - u >2014-09-09 17:38:03,731 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Verify: 12/113: gettext.x86_64 0:0.17-16.el6 - u. >2014-09-09 17:38:03,808 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum Verify: 13/113: gpxe-roms-qemu.noarch 0:0.9.7-6.10.el6 - u >2014-09-09 17:38:03,824 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Verify: 13/113: gpxe-roms-qemu.noarch 0:0.9.7-6.10.el6 - u. >2014-09-09 17:38:03,873 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum Verify: 14/113: netpbm.x86_64 0:10.47.05-11.el6 - u >2014-09-09 17:38:03,891 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Verify: 14/113: netpbm.x86_64 0:10.47.05-11.el6 - u. >2014-09-09 17:38:03,968 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum Verify: 15/113: gnutls.x86_64 0:2.8.5-14.el6_5 - u >2014-09-09 17:38:03,979 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Verify: 15/113: gnutls.x86_64 0:2.8.5-14.el6_5 - u. >2014-09-09 17:38:04,060 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum Verify: 16/113: nc.x86_64 0:1.84-22.el6 - u >2014-09-09 17:38:04,071 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Verify: 16/113: nc.x86_64 0:1.84-22.el6 - u. >2014-09-09 17:38:04,169 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum Verify: 17/113: mom.noarch 0:0.4.1-2.el6 - u >2014-09-09 17:38:04,183 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Verify: 17/113: mom.noarch 0:0.4.1-2.el6 - u. >2014-09-09 17:38:04,237 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum Verify: 18/113: pyparted.x86_64 0:3.4-4.el6 - u >2014-09-09 17:38:04,251 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Verify: 18/113: pyparted.x86_64 0:3.4-4.el6 - u. >2014-09-09 17:38:04,290 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum Verify: 19/113: python-pthreading.noarch 0:0.1.3-3.el6ev - u >2014-09-09 17:38:04,307 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Verify: 19/113: python-pthreading.noarch 0:0.1.3-3.el6ev - u. >2014-09-09 17:38:04,380 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum Verify: 20/113: vdsm-yajsonrpc.noarch 0:4.16.3-2.el6 - u >2014-09-09 17:38:04,395 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Verify: 20/113: vdsm-yajsonrpc.noarch 0:4.16.3-2.el6 - u. >2014-09-09 17:38:04,456 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum Verify: 21/113: fuse.x86_64 0:2.8.3-4.el6 - u >2014-09-09 17:38:04,471 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Verify: 21/113: fuse.x86_64 0:2.8.3-4.el6 - u. >2014-09-09 17:38:04,558 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum Verify: 22/113: gnutls-utils.x86_64 0:2.8.5-14.el6_5 - u >2014-09-09 17:38:04,571 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Verify: 22/113: gnutls-utils.x86_64 0:2.8.5-14.el6_5 - u. >2014-09-09 17:38:04,607 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum Verify: 23/113: libsemanage-python.x86_64 0:2.0.43-4.2.el6 - u >2014-09-09 17:38:04,620 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Verify: 23/113: libsemanage-python.x86_64 0:2.0.43-4.2.el6 - u. >2014-09-09 17:38:04,701 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum Verify: 24/113: libvirt.x86_64 0:0.10.2-29.el6_5.12 - u >2014-09-09 17:38:04,715 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Verify: 24/113: libvirt.x86_64 0:0.10.2-29.el6_5.12 - u. >2014-09-09 17:38:04,802 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum Verify: 25/113: python-decorator.noarch 0:3.0.1-3.1.el6 - u >2014-09-09 17:38:04,830 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Verify: 25/113: python-decorator.noarch 0:3.0.1-3.1.el6 - u. >2014-09-09 17:38:04,886 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum Verify: 26/113: flac.x86_64 0:1.2.1-6.1.el6 - u >2014-09-09 17:38:04,900 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Verify: 26/113: flac.x86_64 0:1.2.1-6.1.el6 - u. >2014-09-09 17:38:04,987 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum Verify: 27/113: febootstrap-supermin-helper.x86_64 0:3.21-4.el6 - u >2014-09-09 17:38:05,001 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Verify: 27/113: febootstrap-supermin-helper.x86_64 0:3.21-4.el6 - u. >2014-09-09 17:38:05,048 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum Verify: 28/113: vdsm-python-zombiereaper.noarch 0:4.16.3-2.el6 - u >2014-09-09 17:38:05,062 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Verify: 28/113: vdsm-python-zombiereaper.noarch 0:4.16.3-2.el6 - u. >2014-09-09 17:38:05,107 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum Verify: 29/113: kpartx.x86_64 0:0.4.9-72.el6_5.3 - u >2014-09-09 17:38:05,125 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Verify: 29/113: kpartx.x86_64 0:0.4.9-72.el6_5.3 - u. >2014-09-09 17:38:05,180 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum Verify: 30/113: libasyncns.x86_64 0:0.8-1.1.el6 - u >2014-09-09 17:38:05,194 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Verify: 30/113: libasyncns.x86_64 0:0.8-1.1.el6 - u. >2014-09-09 17:38:05,273 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum Verify: 31/113: radvd.x86_64 0:1.6-1.el6 - u >2014-09-09 17:38:05,284 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Verify: 31/113: radvd.x86_64 0:1.6-1.el6 - u. >2014-09-09 17:38:05,365 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum Verify: 32/113: usbredir.x86_64 0:0.5.1-1.el6 - u >2014-09-09 17:38:05,379 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Verify: 32/113: usbredir.x86_64 0:0.5.1-1.el6 - u. >2014-09-09 17:38:05,459 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum Verify: 33/113: setools-libs.x86_64 0:3.3.7-4.el6 - u >2014-09-09 17:38:05,476 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Verify: 33/113: setools-libs.x86_64 0:3.3.7-4.el6 - u. >2014-09-09 17:38:05,515 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum Verify: 34/113: vdsm-cli.noarch 0:4.16.3-2.el6 - u >2014-09-09 17:38:05,531 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Verify: 34/113: vdsm-cli.noarch 0:4.16.3-2.el6 - u. >2014-09-09 17:38:05,602 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum Verify: 35/113: scrub.x86_64 0:2.2-1.el6 - u >2014-09-09 17:38:05,615 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Verify: 35/113: scrub.x86_64 0:2.2-1.el6 - u. >2014-09-09 17:38:05,678 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum Verify: 36/113: genisoimage.x86_64 0:1.1.9-12.el6 - u >2014-09-09 17:38:05,688 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Verify: 36/113: genisoimage.x86_64 0:1.1.9-12.el6 - u. >2014-09-09 17:38:05,778 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum Verify: 37/113: vdsm-python.x86_64 0:4.16.3-2.el6 - u >2014-09-09 17:38:05,790 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Verify: 37/113: vdsm-python.x86_64 0:4.16.3-2.el6 - u. >2014-09-09 17:38:05,854 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum Verify: 38/113: btrfs-progs.x86_64 0:0.20-0.2.git91d9eec.el6 - u >2014-09-09 17:38:05,865 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Verify: 38/113: btrfs-progs.x86_64 0:0.20-0.2.git91d9eec.el6 - u. >2014-09-09 17:38:05,923 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum Verify: 39/113: audit.x86_64 0:2.2-4.el6_5 - u >2014-09-09 17:38:05,934 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Verify: 39/113: audit.x86_64 0:2.2-4.el6_5 - u. >2014-09-09 17:38:06,030 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum Verify: 40/113: libtirpc.x86_64 0:0.2.1-6.el6_5.2 - u >2014-09-09 17:38:06,041 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Verify: 40/113: libtirpc.x86_64 0:0.2.1-6.el6_5.2 - u. >2014-09-09 17:38:06,114 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum Verify: 41/113: net-snmp-utils.x86_64 1:5.5-49.el6_5.2 - u >2014-09-09 17:38:06,128 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Verify: 41/113: net-snmp-utils.x86_64 1:5.5-49.el6_5.2 - u. >2014-09-09 17:38:06,207 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum Verify: 42/113: rpcbind.x86_64 0:0.2.0-11.el6 - u >2014-09-09 17:38:06,223 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Verify: 42/113: rpcbind.x86_64 0:0.2.0-11.el6 - u. >2014-09-09 17:38:06,273 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum Verify: 43/113: python-cpopen.x86_64 0:1.3-4.el6_5 - u >2014-09-09 17:38:06,288 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Verify: 43/113: python-cpopen.x86_64 0:1.3-4.el6_5 - u. >2014-09-09 17:38:06,376 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum Verify: 44/113: qemu-kvm-rhev.x86_64 2:0.12.1.2-2.415.el6_5.14 - u >2014-09-09 17:38:06,390 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Verify: 44/113: qemu-kvm-rhev.x86_64 2:0.12.1.2-2.415.el6_5.14 - u. >2014-09-09 17:38:06,448 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum Verify: 45/113: numad.x86_64 0:0.5-10.20140620git.el6_5 - u >2014-09-09 17:38:06,463 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Verify: 45/113: numad.x86_64 0:0.5-10.20140620git.el6_5 - u. >2014-09-09 17:38:06,511 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum Verify: 46/113: keyutils.x86_64 0:1.4-4.el6 - u >2014-09-09 17:38:06,524 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Verify: 46/113: keyutils.x86_64 0:1.4-4.el6 - u. >2014-09-09 17:38:06,603 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum Verify: 47/113: augeas.x86_64 0:1.0.0-5.el6_5.1 - u >2014-09-09 17:38:06,616 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Verify: 47/113: augeas.x86_64 0:1.0.0-5.el6_5.1 - u. >2014-09-09 17:38:06,688 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum Verify: 48/113: pexpect.noarch 0:2.3-6.el6 - u >2014-09-09 17:38:06,699 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Verify: 48/113: pexpect.noarch 0:2.3-6.el6 - u. >2014-09-09 17:38:06,780 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum Verify: 49/113: ebtables.x86_64 0:2.0.9-6.el6 - u >2014-09-09 17:38:06,797 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Verify: 49/113: ebtables.x86_64 0:2.0.9-6.el6 - u. >2014-09-09 17:38:06,840 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum Verify: 50/113: ghostscript.x86_64 0:8.70-19.el6 - u >2014-09-09 17:38:06,854 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Verify: 50/113: ghostscript.x86_64 0:8.70-19.el6 - u. >2014-09-09 17:38:06,915 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum Verify: 51/113: libguestfs-tools-c.x86_64 1:1.20.11-2.el6 - u >2014-09-09 17:38:06,928 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Verify: 51/113: libguestfs-tools-c.x86_64 1:1.20.11-2.el6 - u. >2014-09-09 17:38:07,007 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum Verify: 52/113: nfs-utils-lib.x86_64 0:1.1.5-6.el6_5 - u >2014-09-09 17:38:07,024 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Verify: 52/113: nfs-utils-lib.x86_64 0:1.1.5-6.el6_5 - u. >2014-09-09 17:38:07,082 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum Verify: 53/113: libXfont.x86_64 0:1.4.5-3.el6_5 - u >2014-09-09 17:38:07,096 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Verify: 53/113: libXfont.x86_64 0:1.4.5-3.el6_5 - u. >2014-09-09 17:38:07,167 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum Verify: 54/113: setools-libs-python.x86_64 0:3.3.7-4.el6 - u >2014-09-09 17:38:07,184 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Verify: 54/113: setools-libs-python.x86_64 0:3.3.7-4.el6 - u. >2014-09-09 17:38:07,232 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum Verify: 55/113: vdsm-jsonrpc.noarch 0:4.16.3-2.el6 - u >2014-09-09 17:38:07,250 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Verify: 55/113: vdsm-jsonrpc.noarch 0:4.16.3-2.el6 - u. >2014-09-09 17:38:07,336 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum Verify: 56/113: fence-agents.x86_64 0:3.1.5-35.el6_5.4 - u >2014-09-09 17:38:07,351 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Verify: 56/113: fence-agents.x86_64 0:3.1.5-35.el6_5.4 - u. >2014-09-09 17:38:07,421 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum Verify: 57/113: python-inotify.noarch 0:0.9.1-1.el6ev - u >2014-09-09 17:38:07,436 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Verify: 57/113: python-inotify.noarch 0:0.9.1-1.el6ev - u. >2014-09-09 17:38:07,488 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum Verify: 58/113: libevent.x86_64 0:1.4.13-4.el6 - u >2014-09-09 17:38:07,498 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Verify: 58/113: libevent.x86_64 0:1.4.13-4.el6 - u. >2014-09-09 17:38:07,572 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum Verify: 59/113: urw-fonts.noarch 0:2.4-10.el6 - u >2014-09-09 17:38:07,583 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Verify: 59/113: urw-fonts.noarch 0:2.4-10.el6 - u. >2014-09-09 17:38:07,632 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum Verify: 60/113: telnet.x86_64 1:0.17-47.el6_3.1 - u >2014-09-09 17:38:07,645 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Verify: 60/113: telnet.x86_64 1:0.17-47.el6_3.1 - u. >2014-09-09 17:38:07,732 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum Verify: 61/113: audit-libs-python.x86_64 0:2.2-4.el6_5 - u >2014-09-09 17:38:07,746 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Verify: 61/113: audit-libs-python.x86_64 0:2.2-4.el6_5 - u. >2014-09-09 17:38:07,799 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum Verify: 62/113: netcf-libs.x86_64 0:0.1.9-4.el6_5.2 - u >2014-09-09 17:38:07,816 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Verify: 62/113: netcf-libs.x86_64 0:0.1.9-4.el6_5.2 - u. >2014-09-09 17:38:07,865 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum Verify: 63/113: python-suds.noarch 0:0.4.1-3.el6 - u >2014-09-09 17:38:07,880 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Verify: 63/113: python-suds.noarch 0:0.4.1-3.el6 - u. >2014-09-09 17:38:07,959 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum Verify: 64/113: libcgroup.x86_64 0:0.40.rc1-5.el6_5.1 - u >2014-09-09 17:38:07,973 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Verify: 64/113: libcgroup.x86_64 0:0.40.rc1-5.el6_5.1 - u. >2014-09-09 17:38:08,043 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum Verify: 65/113: seabios.x86_64 0:0.6.1.2-28.el6 - u >2014-09-09 17:38:08,057 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Verify: 65/113: seabios.x86_64 0:0.6.1.2-28.el6 - u. >2014-09-09 17:38:08,127 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum Verify: 66/113: libogg.x86_64 2:1.1.4-2.1.el6 - u >2014-09-09 17:38:08,140 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Verify: 66/113: libogg.x86_64 2:1.1.4-2.1.el6 - u. >2014-09-09 17:38:08,182 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum Verify: 67/113: libXtst.x86_64 0:1.2.1-2.el6 - u >2014-09-09 17:38:08,196 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Verify: 67/113: libXtst.x86_64 0:1.2.1-2.el6 - u. >2014-09-09 17:38:08,240 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum Verify: 68/113: fuse-libs.x86_64 0:2.8.3-4.el6 - u >2014-09-09 17:38:08,255 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Verify: 68/113: fuse-libs.x86_64 0:2.8.3-4.el6 - u. >2014-09-09 17:38:08,322 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum Verify: 69/113: libsndfile.x86_64 0:1.0.20-5.el6 - u >2014-09-09 17:38:08,335 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Verify: 69/113: libsndfile.x86_64 0:1.0.20-5.el6 - u. >2014-09-09 17:38:08,390 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum Verify: 70/113: perl-Net-Telnet.noarch 0:3.03-11.el6 - u >2014-09-09 17:38:08,402 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Verify: 70/113: perl-Net-Telnet.noarch 0:3.03-11.el6 - u. >2014-09-09 17:38:08,473 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum Verify: 71/113: net-snmp-libs.x86_64 1:5.5-49.el6_5.2 - u >2014-09-09 17:38:08,485 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Verify: 71/113: net-snmp-libs.x86_64 1:5.5-49.el6_5.2 - u. >2014-09-09 17:38:08,566 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum Verify: 72/113: policycoreutils-python.x86_64 0:2.0.83-19.39.el6 - u >2014-09-09 17:38:08,579 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Verify: 72/113: policycoreutils-python.x86_64 0:2.0.83-19.39.el6 - u. >2014-09-09 17:38:08,658 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum Verify: 73/113: yajl.x86_64 0:1.0.7-3.el6 - u >2014-09-09 17:38:08,672 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Verify: 73/113: yajl.x86_64 0:1.0.7-3.el6 - u. >2014-09-09 17:38:08,759 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum Verify: 74/113: cvs.x86_64 0:1.11.23-16.el6 - u >2014-09-09 17:38:08,773 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Verify: 74/113: cvs.x86_64 0:1.11.23-16.el6 - u. >2014-09-09 17:38:08,815 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum Verify: 75/113: nfs-utils.x86_64 1:1.2.3-39.el6_5.3 - u >2014-09-09 17:38:08,829 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Verify: 75/113: nfs-utils.x86_64 1:1.2.3-39.el6_5.3 - u. >2014-09-09 17:38:08,911 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum Verify: 76/113: audit-libs.x86_64 0:2.2-4.el6_5 - u >2014-09-09 17:38:08,925 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Verify: 76/113: audit-libs.x86_64 0:2.2-4.el6_5 - u. >2014-09-09 17:38:08,965 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum Verify: 77/113: sanlock-lib.x86_64 0:2.8-1.el6 - u >2014-09-09 17:38:08,979 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Verify: 77/113: sanlock-lib.x86_64 0:2.8-1.el6 - u. >2014-09-09 17:38:09,054 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum Verify: 78/113: python-ioprocess.noarch 0:0.12.0-2.el6 - u >2014-09-09 17:38:09,067 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Verify: 78/113: python-ioprocess.noarch 0:0.12.0-2.el6 - u. >2014-09-09 17:38:09,107 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum Verify: 79/113: netpbm-progs.x86_64 0:10.47.05-11.el6 - u >2014-09-09 17:38:09,119 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Verify: 79/113: netpbm-progs.x86_64 0:10.47.05-11.el6 - u. >2014-09-09 17:38:09,157 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum Verify: 80/113: python-argparse.noarch 0:1.2.1-2.1.el6 - u >2014-09-09 17:38:09,172 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Verify: 80/113: python-argparse.noarch 0:1.2.1-2.1.el6 - u. >2014-09-09 17:38:09,232 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum Verify: 81/113: libvorbis.x86_64 1:1.2.3-4.el6_2.1 - u >2014-09-09 17:38:09,246 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Verify: 81/113: libvorbis.x86_64 1:1.2.3-4.el6_2.1 - u. >2014-09-09 17:38:09,290 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum Verify: 82/113: sg3_utils.x86_64 0:1.28-5.el6 - u >2014-09-09 17:38:09,304 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Verify: 82/113: sg3_utils.x86_64 0:1.28-5.el6 - u. >2014-09-09 17:38:09,367 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum Verify: 83/113: spice-server.x86_64 0:0.12.4-6.el6_5.2 - u >2014-09-09 17:38:09,380 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Verify: 83/113: spice-server.x86_64 0:0.12.4-6.el6_5.2 - u. >2014-09-09 17:38:09,432 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum Verify: 84/113: glusterfs.x86_64 0:3.5.2-1.el6 - u >2014-09-09 17:38:09,446 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Verify: 84/113: glusterfs.x86_64 0:3.5.2-1.el6 - u. >2014-09-09 17:38:09,518 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum Verify: 85/113: perl-hivex.x86_64 0:1.3.3-4.2.el6 - u >2014-09-09 17:38:09,532 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Verify: 85/113: perl-hivex.x86_64 0:1.3.3-4.2.el6 - u. >2014-09-09 17:38:09,619 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum Verify: 86/113: pulseaudio-libs.x86_64 0:0.9.21-14.el6_3 - u >2014-09-09 17:38:09,630 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Verify: 86/113: pulseaudio-libs.x86_64 0:0.9.21-14.el6_3 - u. >2014-09-09 17:38:09,674 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum Verify: 87/113: python-ordereddict.noarch 0:1.1-3.el6ev - u >2014-09-09 17:38:09,685 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Verify: 87/113: python-ordereddict.noarch 0:1.1-3.el6ev - u. >2014-09-09 17:38:09,754 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum Verify: 88/113: libfontenc.x86_64 0:1.0.5-2.el6 - u >2014-09-09 17:38:09,772 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Verify: 88/113: libfontenc.x86_64 0:1.0.5-2.el6 - u. >2014-09-09 17:38:09,821 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum Verify: 89/113: xorg-x11-font-utils.x86_64 1:7.2-11.el6 - u >2014-09-09 17:38:09,835 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Verify: 89/113: xorg-x11-font-utils.x86_64 1:7.2-11.el6 - u. >2014-09-09 17:38:09,873 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum Verify: 90/113: ioprocess.x86_64 0:0.12.0-2.el6 - u >2014-09-09 17:38:09,888 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Verify: 90/113: ioprocess.x86_64 0:0.12.0-2.el6 - u. >2014-09-09 17:38:09,958 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum Verify: 91/113: lzop.x86_64 0:1.02-0.9.rc1.el6 - u >2014-09-09 17:38:09,971 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Verify: 91/113: lzop.x86_64 0:1.02-0.9.rc1.el6 - u. >2014-09-09 17:38:10,016 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum Verify: 92/113: libvirt-client.x86_64 0:0.10.2-29.el6_5.12 - u >2014-09-09 17:38:10,031 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Verify: 92/113: libvirt-client.x86_64 0:0.10.2-29.el6_5.12 - u. >2014-09-09 17:38:10,101 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum Verify: 93/113: vdsm-xmlrpc.noarch 0:4.16.3-2.el6 - u >2014-09-09 17:38:10,114 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Verify: 93/113: vdsm-xmlrpc.noarch 0:4.16.3-2.el6 - u. >2014-09-09 17:38:10,157 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum Verify: 94/113: sanlock-python.x86_64 0:2.8-1.el6 - u >2014-09-09 17:38:10,170 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Verify: 94/113: sanlock-python.x86_64 0:2.8-1.el6 - u. >2014-09-09 17:38:10,224 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum Verify: 95/113: glusterfs-api.x86_64 0:3.5.2-1.el6 - u >2014-09-09 17:38:10,237 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Verify: 95/113: glusterfs-api.x86_64 0:3.5.2-1.el6 - u. >2014-09-09 17:38:10,313 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum Verify: 96/113: dnsmasq.x86_64 0:2.48-13.el6 - u >2014-09-09 17:38:10,330 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Verify: 96/113: dnsmasq.x86_64 0:2.48-13.el6 - u. >2014-09-09 17:38:10,387 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum Verify: 97/113: python-netaddr.noarch 0:0.7.5-4.el6 - u >2014-09-09 17:38:10,404 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Verify: 97/113: python-netaddr.noarch 0:0.7.5-4.el6 - u. >2014-09-09 17:38:10,463 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum Verify: 98/113: libgomp.x86_64 0:4.4.7-4.el6 - u >2014-09-09 17:38:10,476 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Verify: 98/113: libgomp.x86_64 0:4.4.7-4.el6 - u. >2014-09-09 17:38:10,515 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum Verify: 99/113: device-mapper-multipath.x86_64 0:0.4.9-72.el6_5.3 - u >2014-09-09 17:38:10,529 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Verify: 99/113: device-mapper-multipath.x86_64 0:0.4.9-72.el6_5.3 - u. >2014-09-09 17:38:10,606 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum Verify: 100/113: ipmitool.x86_64 0:1.8.11-16.el6 - u >2014-09-09 17:38:10,620 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Verify: 100/113: ipmitool.x86_64 0:1.8.11-16.el6 - u. >2014-09-09 17:38:10,674 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum Verify: 101/113: cyrus-sasl-md5.x86_64 0:2.1.23-13.el6_3.1 - u >2014-09-09 17:38:10,686 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Verify: 101/113: cyrus-sasl-md5.x86_64 0:2.1.23-13.el6_3.1 - u. >2014-09-09 17:38:10,732 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum Verify: 102/113: augeas-libs.x86_64 0:1.0.0-5.el6_5.1 - u >2014-09-09 17:38:10,743 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Verify: 102/113: augeas-libs.x86_64 0:1.0.0-5.el6_5.1 - u. >2014-09-09 17:38:10,818 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum Verify: 103/113: lm_sensors-libs.x86_64 0:3.1.1-17.el6 - u >2014-09-09 17:38:10,833 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Verify: 103/113: lm_sensors-libs.x86_64 0:3.1.1-17.el6 - u. >2014-09-09 17:38:10,893 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum Verify: 104/113: libguestfs.x86_64 1:1.20.11-2.el6 - u >2014-09-09 17:38:10,907 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Verify: 104/113: libguestfs.x86_64 1:1.20.11-2.el6 - u. >2014-09-09 17:38:10,985 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum Verify: 105/113: glusterfs-libs.x86_64 0:3.5.2-1.el6 - u >2014-09-09 17:38:11,001 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Verify: 105/113: glusterfs-libs.x86_64 0:3.5.2-1.el6 - u. >2014-09-09 17:38:11,070 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum Verify: 106/113: libgssglue.x86_64 0:0.1-11.el6 - u >2014-09-09 17:38:11,084 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Verify: 106/113: libgssglue.x86_64 0:0.1-11.el6 - u. >2014-09-09 17:38:11,132 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum Verify: 107/113: libvirt-lock-sanlock.x86_64 0:0.10.2-29.el6_5.12 - u >2014-09-09 17:38:11,146 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Verify: 107/113: libvirt-lock-sanlock.x86_64 0:0.10.2-29.el6_5.12 - u. >2014-09-09 17:38:11,204 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum Verify: 108/113: vdsm.x86_64 0:4.16.3-2.el6 - u >2014-09-09 17:38:11,218 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Verify: 108/113: vdsm.x86_64 0:4.16.3-2.el6 - u. >2014-09-09 17:38:11,257 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum Verify: 109/113: libvirt-python.x86_64 0:0.10.2-29.el6_5.12 - u >2014-09-09 17:38:11,271 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Verify: 109/113: libvirt-python.x86_64 0:0.10.2-29.el6_5.12 - u. >2014-09-09 17:38:11,356 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum Verify: 110/113: kpartx.x86_64 0:0.4.9-72.el6 - ud >2014-09-09 17:38:11,372 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Verify: 110/113: kpartx.x86_64 0:0.4.9-72.el6 - ud. >2014-09-09 17:38:11,372 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum Verify: 111/113: audit.x86_64 0:2.2-2.el6 - ud >2014-09-09 17:38:11,391 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Verify: 111/113: audit.x86_64 0:2.2-2.el6 - ud. >2014-09-09 17:38:11,391 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum Verify: 112/113: audit-libs.x86_64 0:2.2-2.el6 - ud >2014-09-09 17:38:11,410 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Verify: 112/113: audit-libs.x86_64 0:2.2-2.el6 - ud. >2014-09-09 17:38:11,410 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Yum Verify: 113/113: gnutls.x86_64 0:2.8.5-10.el6_4.2 - ud >2014-09-09 17:38:11,424 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Verify: 113/113: gnutls.x86_64 0:2.8.5-10.el6_4.2 - ud. >2014-09-09 17:38:11,515 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Stage: Misc configuration >2014-09-09 17:38:11,526 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Stage: Misc configuration. >2014-09-09 17:38:11,622 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Setting time >2014-09-09 17:38:11,636 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Setting time. >2014-09-09 17:38:13,488 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Enrolling certificate >2014-09-09 17:38:13,502 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Enrolling certificate. >2014-09-09 17:38:14,789 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Stage: Transaction commit >2014-09-09 17:38:14,807 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Stage: Transaction commit. >2014-09-09 17:38:14,954 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Stage: Closing up >2014-09-09 17:38:14,968 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Stage: Closing up. >2014-09-09 17:38:15,908 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Stopping libvirtd >2014-09-09 17:38:15,925 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Stopping libvirtd. >2014-09-09 17:38:15,950 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Starting vdsm >2014-09-09 17:38:15,962 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Starting vdsm. >2014-09-09 17:38:19,192 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Stage: Pre-termination >2014-09-09 17:38:19,207 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Stage: Pre-termination. >2014-09-09 17:38:19,229 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Retrieving installation logs to: '/var/log/ovirt-engine/host-deploy/ovirt-20140909173819-dell-r210ii-16.rhev.lab.eng.brq.redhat.com-6093dd15.log' >2014-09-09 17:38:19,246 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Retrieving installation logs to: '/var/log/ovirt-engine/host-deploy/ovirt-20140909173819-dell-r210ii-16.rhev.lab.eng.brq.redhat.com-6093dd15.log'. >2014-09-09 17:38:19,670 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Stage: Termination >2014-09-09 17:38:19,684 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6093dd15, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Stage: Termination. >2014-09-09 17:38:19,841 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to dell-r210ii-16.rhev.lab.eng.brq.redhat.com/10.34.62.208 >2014-09-09 17:38:20,070 INFO [org.ovirt.engine.core.bll.network.NetworkConfigurator] (org.ovirt.thread.pool-6-thread-36) [6093dd15] Engine managed to communicate with VDSM agent on host rack >2014-09-09 17:38:20,780 INFO [org.ovirt.engine.core.bll.network.host.SetupNetworksCommand] (org.ovirt.thread.pool-6-thread-36) [7bcb399f] Running command: SetupNetworksCommand internal: true. Entities affected : ID: 7ebdb254-efb6-480b-8c4e-a7dbdc759ed4 Type: VDSAction group CONFIGURE_HOST_NETWORK with role type ADMIN >2014-09-09 17:38:20,792 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetupNetworksVDSCommand] (org.ovirt.thread.pool-6-thread-36) [7bcb399f] START, SetupNetworksVDSCommand(HostName = rack, HostId = 7ebdb254-efb6-480b-8c4e-a7dbdc759ed4, force=false, checkConnectivity=true, conectivityTimeout=120, > networks=[rhevm {id=00000000-0000-0000-0000-000000000009, description=Management Network, comment=null, subnet=null, gateway=null, type=null, vlanId=null, stp=false, dataCenterId=00000002-0002-0002-0002-0000000001ad, mtu=0, vmNetwork=true, cluster=NetworkCluster {id={clusterId=null, networkId=null}, status=OPERATIONAL, display=true, required=true, migration=true}, providedBy=null, label=null, qosId=null}], > bonds=[], > interfaces=[em1 {id=c97613e8-b860-495b-9207-7715533de9eb, vdsId=7ebdb254-efb6-480b-8c4e-a7dbdc759ed4, macAddress=d4:ae:52:c7:0d:8b, networkName=rhevm, baseInterface=null, vlanId=null, bonded=null, bondName=null, bondOptions=null, bootProtocol=DHCP, address=10.34.62.208, subnet=255.255.252.0, gateway=null, mtu=1500, bridged=false, speed=1000, type=0, networkImplementationDetails=null, qos=null qosOverridden=false, labels=null, customProperties=null}, > em2 {id=ca117a0b-8f32-4c38-8b4c-aa15bd19f786, vdsId=7ebdb254-efb6-480b-8c4e-a7dbdc759ed4, macAddress=d4:ae:52:c7:0d:8c, networkName=null, baseInterface=null, vlanId=null, bonded=null, bondName=null, bondOptions=null, bootProtocol=DHCP, address=, subnet=, gateway=null, mtu=1500, bridged=false, speed=0, type=0, networkImplementationDetails=null, qos=null qosOverridden=false, labels=null, customProperties=null}], > removedNetworks=[], > removedBonds=[]), log id: 4c9c1b2b >2014-09-09 17:38:20,807 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetupNetworksVDSCommand] (org.ovirt.thread.pool-6-thread-36) [7bcb399f] FINISH, SetupNetworksVDSCommand, log id: 4c9c1b2b >2014-09-09 17:41:21,122 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SetupNetworksVDSCommand] (org.ovirt.thread.pool-6-thread-36) [7bcb399f] org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues' >2014-09-09 17:41:21,123 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SetupNetworksVDSCommand] (org.ovirt.thread.pool-6-thread-36) [7bcb399f] Command SetupNetworksVDSCommand(HostName = rack, HostId = 7ebdb254-efb6-480b-8c4e-a7dbdc759ed4, force=false, checkConnectivity=true, conectivityTimeout=120, > networks=[rhevm {id=00000000-0000-0000-0000-000000000009, description=Management Network, comment=null, subnet=null, gateway=null, type=null, vlanId=null, stp=false, dataCenterId=00000002-0002-0002-0002-0000000001ad, mtu=0, vmNetwork=true, cluster=NetworkCluster {id={clusterId=null, networkId=null}, status=OPERATIONAL, display=true, required=true, migration=true}, providedBy=null, label=null, qosId=null}], > bonds=[], > interfaces=[em1 {id=c97613e8-b860-495b-9207-7715533de9eb, vdsId=7ebdb254-efb6-480b-8c4e-a7dbdc759ed4, macAddress=d4:ae:52:c7:0d:8b, networkName=rhevm, baseInterface=null, vlanId=null, bonded=null, bondName=null, bondOptions=null, bootProtocol=DHCP, address=10.34.62.208, subnet=255.255.252.0, gateway=null, mtu=1500, bridged=false, speed=1000, type=0, networkImplementationDetails=null, qos=null qosOverridden=false, labels=null, customProperties=null}, > em2 {id=ca117a0b-8f32-4c38-8b4c-aa15bd19f786, vdsId=7ebdb254-efb6-480b-8c4e-a7dbdc759ed4, macAddress=d4:ae:52:c7:0d:8c, networkName=null, baseInterface=null, vlanId=null, bonded=null, bondName=null, bondOptions=null, bootProtocol=DHCP, address=, subnet=, gateway=null, mtu=1500, bridged=false, speed=0, type=0, networkImplementationDetails=null, qos=null qosOverridden=false, labels=null, customProperties=null}], > removedNetworks=[], > removedBonds=[]) execution failed. Exception: VDSNetworkException: VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues' >2014-09-09 17:41:21,124 ERROR [org.ovirt.engine.core.bll.network.host.SetupNetworksCommand] (org.ovirt.thread.pool-6-thread-36) [7bcb399f] Command org.ovirt.engine.core.bll.network.host.SetupNetworksCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues' (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-09-09 17:41:21,140 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-36) [7bcb399f] Failed to configure management network: Failed to configure management network on host rack due to setup networks failure. >2014-09-09 17:41:21,144 ERROR [org.ovirt.engine.core.bll.InstallVdsInternalCommand] (org.ovirt.thread.pool-6-thread-36) [7bcb399f] Host installation failed for host 7ebdb254-efb6-480b-8c4e-a7dbdc759ed4, rack.: org.ovirt.engine.core.bll.VdsCommand$VdsInstallException: Failed to configure management network on the host > at org.ovirt.engine.core.bll.InstallVdsInternalCommand.configureManagementNetwork(InstallVdsInternalCommand.java:206) [bll.jar:] > at org.ovirt.engine.core.bll.InstallVdsInternalCommand.installHost(InstallVdsInternalCommand.java:164) [bll.jar:] > at org.ovirt.engine.core.bll.InstallVdsInternalCommand.executeCommand(InstallVdsInternalCommand.java:78) [bll.jar:] > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1167) [bll.jar:] > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1298) [bll.jar:] > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1920) [bll.jar:] > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) [utils.jar:] > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) [utils.jar:] > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1319) [bll.jar:] > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:344) [bll.jar:] > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:430) [bll.jar:] > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:411) [bll.jar:] > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:616) [bll.jar:] > at sun.reflect.GeneratedMethodAccessor78.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptor.java:52) [jboss-as-ee.jar:7.4.0.Final-redhat-19] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:63) [jboss-as-ee.jar:7.4.0.Final-redhat-19] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:374) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.as.weld.ejb.Jsr299BindingsInterceptor.delegateInterception(Jsr299BindingsInterceptor.java:74) [jboss-as-weld.jar:7.4.0.Final-redhat-19] > at org.jboss.as.weld.ejb.Jsr299BindingsInterceptor.doMethodInterception(Jsr299BindingsInterceptor.java:84) [jboss-as-weld.jar:7.4.0.Final-redhat-19] > at org.jboss.as.weld.ejb.Jsr299BindingsInterceptor.processInvocation(Jsr299BindingsInterceptor.java:97) [jboss-as-weld.jar:7.4.0.Final-redhat-19] > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:63) [jboss-as-ee.jar:7.4.0.Final-redhat-19] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:63) [jboss-as-ee.jar:7.4.0.Final-redhat-19] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) [jboss-as-ejb3.jar:7.4.0.Final-redhat-19] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.as.weld.ejb.EjbRequestScopeActivationInterceptor.processInvocation(EjbRequestScopeActivationInterceptor.java:93) [jboss-as-weld.jar:7.4.0.Final-redhat-19] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) [jboss-as-ee.jar:7.4.0.Final-redhat-19] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) [jboss-as-ejb3.jar:7.4.0.Final-redhat-19] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) [jboss-as-ejb3.jar:7.4.0.Final-redhat-19] > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) [jboss-as-ejb3.jar:7.4.0.Final-redhat-19] > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) [jboss-as-ejb3.jar:7.4.0.Final-redhat-19] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) [jboss-as-ejb3.jar:7.4.0.Final-redhat-19] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) [jboss-as-ejb3.jar:7.4.0.Final-redhat-19] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) [jboss-as-ejb3.jar:7.4.0.Final-redhat-19] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) [jboss-as-ee.jar:7.4.0.Final-redhat-19] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) [jboss-as-ee.jar:7.4.0.Final-redhat-19] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:185) [jboss-as-ee.jar:7.4.0.Final-redhat-19] > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) [jboss-as-ee.jar:7.4.0.Final-redhat-19] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:73) [jboss-as-ee.jar:7.4.0.Final-redhat-19] > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) [bll.jar:] > at org.ovirt.engine.core.bll.CommandBase.runInternalAction(CommandBase.java:2249) [bll.jar:] > at org.ovirt.engine.core.bll.AddVdsCommand$3.run(AddVdsCommand.java:195) [bll.jar:] > at org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil$InternalWrapperRunnable.run(ThreadPoolUtil.java:90) [utils.jar:] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_65] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_65] > at java.lang.Thread.run(Thread.java:745) [rt.jar:1.7.0_65] >Caused by: org.ovirt.engine.core.bll.network.NetworkConfigurator$NetworkConfiguratorException: Failed to configure management network > at org.ovirt.engine.core.bll.network.NetworkConfigurator.configureManagementNetwork(NetworkConfigurator.java:255) [bll.jar:] > at org.ovirt.engine.core.bll.network.NetworkConfigurator.createManagementNetworkIfRequired(NetworkConfigurator.java:82) [bll.jar:] > at org.ovirt.engine.core.bll.InstallVdsInternalCommand.configureManagementNetwork(InstallVdsInternalCommand.java:198) [bll.jar:] > ... 69 more > >2014-09-09 17:41:21,150 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-6-thread-36) [7bcb399f] START, SetVdsStatusVDSCommand(HostName = rack, HostId = 7ebdb254-efb6-480b-8c4e-a7dbdc759ed4, status=NonOperational, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 32132254 >2014-09-09 17:41:21,181 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-6-thread-36) [7bcb399f] FINISH, SetVdsStatusVDSCommand, log id: 32132254 >2014-09-09 17:41:21,200 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-36) [7bcb399f] Correlation ID: 6093dd15, Job ID: 28ad829d-a61d-4309-a303-4bf2b7ff5d54, Call Stack: null, Custom Event ID: -1, Message: Host rack installation failed. Failed to configure management network on the host. >2014-09-09 17:41:21,214 INFO [org.ovirt.engine.core.bll.InstallVdsInternalCommand] (org.ovirt.thread.pool-6-thread-36) [7bcb399f] Lock freed to object EngineLock [exclusiveLocks= key: 7ebdb254-efb6-480b-8c4e-a7dbdc759ed4 value: VDS >, sharedLocks= ] >2014-09-09 17:45:00,042 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-11) Autorecovering 1 hosts >2014-09-09 17:45:00,042 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-11) Autorecovering hosts id: 7ebdb254-efb6-480b-8c4e-a7dbdc759ed4, name : rack >2014-09-09 17:45:00,049 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-11) [7b5e1817] Lock Acquired to object EngineLock [exclusiveLocks= key: 7ebdb254-efb6-480b-8c4e-a7dbdc759ed4 value: VDS >, sharedLocks= ] >2014-09-09 17:45:00,049 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-11) [7b5e1817] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 7ebdb254-efb6-480b-8c4e-a7dbdc759ed4 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2014-09-09 17:45:00,049 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-11) [7b5e1817] Before acquiring lock in order to prevent monitoring for host rack from data-center Default >2014-09-09 17:45:00,049 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-11) [7b5e1817] Lock acquired, from now a monitoring of host will be skipped for host rack from data-center Default >2014-09-09 17:45:00,055 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-11) [7b5e1817] START, SetVdsStatusVDSCommand(HostName = rack, HostId = 7ebdb254-efb6-480b-8c4e-a7dbdc759ed4, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 5101d9e2 >2014-09-09 17:45:00,085 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-11) [7b5e1817] FINISH, SetVdsStatusVDSCommand, log id: 5101d9e2 >2014-09-09 17:45:00,122 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-11) [7b5e1817] Activate finished. Lock released. Monitoring can run now for host rack from data-center Default >2014-09-09 17:45:00,135 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-11) [7b5e1817] Correlation ID: 7b5e1817, Call Stack: null, Custom Event ID: -1, Message: Host rack was autorecovered. >2014-09-09 17:45:00,135 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-11) [7b5e1817] Lock freed to object EngineLock [exclusiveLocks= key: 7ebdb254-efb6-480b-8c4e-a7dbdc759ed4 value: VDS >, sharedLocks= ] >2014-09-09 17:45:01,973 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to dell-r210ii-16.rhev.lab.eng.brq.redhat.com/10.34.62.208 >2014-09-09 17:45:01,975 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.Reactor] (SSL Stomp Reactor) Unable to process messages: java.nio.channels.ClosedChannelException > at sun.nio.ch.SocketChannelImpl.ensureReadOpen(SocketChannelImpl.java:252) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:295) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.SSLEngineNioHelper.read(SSLEngineNioHelper.java:51) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.SSLClient.read(SSLClient.java:81) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.readBuffer(ReactorClient.java:203) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.stomp.StompCommonClient.processIncoming(StompCommonClient.java:90) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.process(ReactorClient.java:144) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.SSLClient.process(SSLClient.java:115) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.processChannels(Reactor.java:86) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:62) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:45:15,474 WARN [org.ovirt.engine.core.bll.network.host.SetupNetworksCommand] (ajp-/127.0.0.1:8702-4) [2872f9da] CanDoAction of action SetupNetworks failed. Reasons:VAR__ACTION__SETUP,VAR__TYPE__NETWORKS,VAR__HOST_STATUS__UP_MAINTENANCE_OR_NON_OPERATIONAL,ACTION_TYPE_FAILED_VDS_STATUS_ILLEGAL >2014-09-09 17:47:23,254 INFO [org.ovirt.engine.core.bll.storage.FenceVdsManualyCommand] (ajp-/127.0.0.1:8702-1) [241f25cb] Lock Acquired to object EngineLock [exclusiveLocks= key: 7ebdb254-efb6-480b-8c4e-a7dbdc759ed4 value: VDS_FENCE >, sharedLocks= ] >2014-09-09 17:47:23,254 WARN [org.ovirt.engine.core.bll.storage.FenceVdsManualyCommand] (ajp-/127.0.0.1:8702-1) [241f25cb] CanDoAction of action FenceVdsManualy failed. Reasons:VAR__ACTION__MANUAL_FENCE,VAR__TYPE__HOST,ACTION_TYPE_FAILED_VDS_NOT_MATCH_VALID_STATUS >2014-09-09 17:47:23,254 INFO [org.ovirt.engine.core.bll.storage.FenceVdsManualyCommand] (ajp-/127.0.0.1:8702-1) [241f25cb] Lock freed to object EngineLock [exclusiveLocks= key: 7ebdb254-efb6-480b-8c4e-a7dbdc759ed4 value: VDS_FENCE >, sharedLocks= ] >2014-09-09 17:47:52,266 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-47) Principal admin::internal synchronized >2014-09-09 17:47:52,600 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-61) Attempting to update VMs/Templates Ovf. >2014-09-09 17:48:04,263 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-30) [5b453678] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@7800906b >2014-09-09 17:48:04,263 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-30) [5b453678] HostName = rack >2014-09-09 17:48:04,263 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-30) [5b453678] Command GetCapabilitiesVDSCommand(HostName = rack, HostId = 7ebdb254-efb6-480b-8c4e-a7dbdc759ed4, vds=Host[rack,7ebdb254-efb6-480b-8c4e-a7dbdc759ed4]) execution failed. Exception: VDSNetworkException: VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues' >2014-09-09 17:48:04,263 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-30) [5b453678] Host rack is not responding. It will stay in Connecting state for a grace period of 60 seconds and after that an attempt to fence the host will be issued. >2014-09-09 17:48:04,289 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-30) [5b453678] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host rack is not responding. It will stay in Connecting state for a grace period of 60 seconds and after that an attempt to fence the host will be issued. >2014-09-09 17:48:04,311 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-30) [5b453678] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues' > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:183) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:16) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] > >2014-09-09 17:48:04,318 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-30) [5b453678] Failed to refresh VDS , vds = 7ebdb254-efb6-480b-8c4e-a7dbdc759ed4 : rack, VDS Network Error, continuing. >VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues' >2014-09-09 17:48:07,356 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to dell-r210ii-16.rhev.lab.eng.brq.redhat.com/10.34.62.208 >2014-09-09 17:48:07,357 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.Reactor] (SSL Stomp Reactor) Unable to process messages: java.nio.channels.ClosedChannelException > at sun.nio.ch.SocketChannelImpl.ensureWriteOpen(SocketChannelImpl.java:265) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:474) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.SSLEngineNioHelper.write(SSLEngineNioHelper.java:104) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.SSLEngineNioHelper.process(SSLEngineNioHelper.java:125) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.SSLClient.pendingOperations(SSLClient.java:74) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.SSLClient.process(SSLClient.java:98) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.processChannels(Reactor.java:86) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:62) [vdsm-jsonrpc-java-client.jar:] > >2014-09-09 17:51:09,827 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-73) Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@194eb0ef >2014-09-09 17:51:09,827 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-73) HostName = rack >2014-09-09 17:51:09,827 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-73) Command GetCapabilitiesVDSCommand(HostName = rack, HostId = 7ebdb254-efb6-480b-8c4e-a7dbdc759ed4, vds=Host[rack,7ebdb254-efb6-480b-8c4e-a7dbdc759ed4]) execution failed. Exception: VDSNetworkException: VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues' >2014-09-09 17:51:09,828 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-73) Host rack is not responding. It will stay in Connecting state for a grace period of 60 seconds and after that an attempt to fence the host will be issued. >2014-09-09 17:51:09,874 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-73) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host rack is not responding. It will stay in Connecting state for a grace period of 60 seconds and after that an attempt to fence the host will be issued. >2014-09-09 17:51:09,874 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-73) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues' > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:183) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:16) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] > >2014-09-09 17:51:09,880 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-73) Failed to refresh VDS , vds = 7ebdb254-efb6-480b-8c4e-a7dbdc759ed4 : rack, VDS Network Error, continuing. >VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues' >2014-09-09 17:51:12,951 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to dell-r210ii-16.rhev.lab.eng.brq.redhat.com/10.34.62.208 >2014-09-09 17:54:15,401 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-45) Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@273ebe7a >2014-09-09 17:54:15,401 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-45) HostName = rack >2014-09-09 17:54:15,401 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-45) Command GetCapabilitiesVDSCommand(HostName = rack, HostId = 7ebdb254-efb6-480b-8c4e-a7dbdc759ed4, vds=Host[rack,7ebdb254-efb6-480b-8c4e-a7dbdc759ed4]) execution failed. Exception: VDSNetworkException: VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues' >2014-09-09 17:54:15,404 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-45) Host rack is not responding. It will stay in Connecting state for a grace period of 60 seconds and after that an attempt to fence the host will be issued. >2014-09-09 17:54:15,425 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-45) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host rack is not responding. It will stay in Connecting state for a grace period of 60 seconds and after that an attempt to fence the host will be issued. >2014-09-09 17:54:15,479 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-45) Server failed to respond, vds_id = 7ebdb254-efb6-480b-8c4e-a7dbdc759ed4, vds_name = rack, vm_count = 0, spm_status = None, non-responsive_timeout (seconds) = 60, error = VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues' >2014-09-09 17:54:15,497 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-45) Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues' > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:183) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:16) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Host rack is non responsive. >2014-09-09 17:54:15,502 INFO [org.ovirt.engine.core.bll.VdsEventListener] (org.ovirt.thread.pool-6-thread-43) ResourceManager::vdsNotResponding entered for Host 7ebdb254-efb6-480b-8c4e-a7dbdc759ed4, dell-r210ii-16.rhev.lab.eng.brq.redhat.com >2014-09-09 17:54:15,552 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-45) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues' > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:183) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:16) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] > >2014-09-09 17:54:15,603 INFO [org.ovirt.engine.core.bll.SshSoftFencingCommand] (org.ovirt.thread.pool-6-thread-43) [138b849d] Running command: SshSoftFencingCommand internal: true. Entities affected : ID: 7ebdb254-efb6-480b-8c4e-a7dbdc759ed4 Type: VDS >2014-09-09 17:54:15,611 INFO [org.ovirt.engine.core.bll.SshSoftFencingCommand] (org.ovirt.thread.pool-6-thread-43) [138b849d] Opening SSH Soft Fencing session on host dell-r210ii-16.rhev.lab.eng.brq.redhat.com >2014-09-09 17:54:15,767 INFO [org.ovirt.engine.core.bll.SshSoftFencingCommand] (org.ovirt.thread.pool-6-thread-43) [138b849d] Executing SSH Soft Fencing command on host dell-r210ii-16.rhev.lab.eng.brq.redhat.com >2014-09-09 17:54:37,726 INFO [org.ovirt.engine.core.bll.MaintenanceNumberOfVdssCommand] (org.ovirt.thread.pool-6-thread-45) [2ce2a17e] Running command: MaintenanceNumberOfVdssCommand internal: false. Entities affected : ID: 7ebdb254-efb6-480b-8c4e-a7dbdc759ed4 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2014-09-09 17:54:37,857 INFO [org.ovirt.engine.core.bll.MaintenanceVdsCommand] (org.ovirt.thread.pool-6-thread-45) [2ce2a17e] Running command: MaintenanceVdsCommand internal: true. Entities affected : ID: 7ebdb254-efb6-480b-8c4e-a7dbdc759ed4 Type: VDS >2014-09-09 17:54:37,863 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-6-thread-45) [2ce2a17e] START, SetVdsStatusVDSCommand(HostName = rack, HostId = 7ebdb254-efb6-480b-8c4e-a7dbdc759ed4, status=Maintenance, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 3a9093a7 >2014-09-09 18:47:52,282 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-80) Principal admin::internal synchronized >2014-09-09 18:47:52,608 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-30) [5b453678] Attempting to update VMs/Templates Ovf. >2014-09-09 19:47:52,296 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-90) [18a5c631] Principal admin::internal synchronized >2014-09-09 19:47:52,612 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-5) Attempting to update VMs/Templates Ovf. >2014-09-09 20:47:52,313 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-23) Principal admin::internal synchronized >2014-09-09 20:47:52,616 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-33) [8ff0b8] Attempting to update VMs/Templates Ovf. >2014-09-09 21:47:52,326 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-54) Principal admin::internal synchronized >2014-09-09 21:47:52,620 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-34) Attempting to update VMs/Templates Ovf. >2014-09-09 22:47:52,338 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-99) Principal admin::internal synchronized >2014-09-09 22:47:52,626 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-79) Attempting to update VMs/Templates Ovf. >2014-09-09 23:47:52,352 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-92) Principal admin::internal synchronized >2014-09-09 23:47:52,630 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-8) [13036fef] Attempting to update VMs/Templates Ovf. >2014-09-10 00:47:52,365 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-16) Principal admin::internal synchronized >2014-09-10 00:47:52,634 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-37) Attempting to update VMs/Templates Ovf. >2014-09-10 01:47:52,376 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-44) Principal admin::internal synchronized >2014-09-10 01:47:52,637 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-52) Attempting to update VMs/Templates Ovf. >2014-09-10 02:47:52,387 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-40) Principal admin::internal synchronized >2014-09-10 02:47:52,640 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-71) Attempting to update VMs/Templates Ovf. >2014-09-10 03:35:35,001 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-34) Start deleteAgedOutAuditLogs >2014-09-10 03:35:35,010 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-55) Start deleteAgedOutCommandEntities >2014-09-10 03:35:35,034 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-34) Finished deleteAgedOutAuditLogs >2014-09-10 03:35:35,038 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-55) Finished deleteAgedOutCommandEntities >2014-09-10 03:47:52,399 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-91) Principal admin::internal synchronized >2014-09-10 03:47:52,644 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-93) Attempting to update VMs/Templates Ovf. >2014-09-10 04:47:52,411 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-12) Principal admin::internal synchronized >2014-09-10 04:47:52,647 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-20) Attempting to update VMs/Templates Ovf. >2014-09-10 05:47:52,428 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-23) Principal admin::internal synchronized >2014-09-10 05:47:52,650 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-51) [5c5dd985] Attempting to update VMs/Templates Ovf. >2014-09-10 06:47:52,444 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-67) Principal admin::internal synchronized >2014-09-10 06:47:52,653 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-81) [24b7b64b] Attempting to update VMs/Templates Ovf. >2014-09-10 07:47:52,464 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-82) Principal admin::internal synchronized >2014-09-10 07:47:52,657 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-93) Attempting to update VMs/Templates Ovf. >2014-09-10 08:47:52,484 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-14) Principal admin::internal synchronized >2014-09-10 08:47:52,659 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-28) Attempting to update VMs/Templates Ovf. >2014-09-10 09:47:52,506 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-68) Principal admin::internal synchronized >2014-09-10 09:47:52,661 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-41) Attempting to update VMs/Templates Ovf. >2014-09-10 10:47:52,522 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-83) [b670564] Principal admin::internal synchronized >2014-09-10 10:47:52,664 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-69) [2f5064a8] Attempting to update VMs/Templates Ovf. >2014-09-10 10:57:22,294 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp-/127.0.0.1:8702-7) [6edc5c44] Running command: LogoutUserCommand internal: false. >2014-09-10 10:57:22,311 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp-/127.0.0.1:8702-9) [29632489] Running command: LogoutUserCommand internal: false. >2014-09-10 10:57:22,312 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp-/127.0.0.1:8702-8) [551c7763] Running command: LogoutUserCommand internal: false. >2014-09-10 10:57:22,321 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp-/127.0.0.1:8702-5) [5967886e] Running command: LogoutUserCommand internal: false. >2014-09-10 10:57:22,324 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp-/127.0.0.1:8702-1) [3fe2303c] Running command: LogoutUserCommand internal: false. >2014-09-10 10:57:22,376 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-8) [551c7763] Correlation ID: 551c7763, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-09-10 10:57:22,384 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp-/127.0.0.1:8702-10) [1742c1b1] Running command: LogoutUserCommand internal: false. >2014-09-10 10:57:22,387 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-7) [6edc5c44] Correlation ID: 6edc5c44, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-09-10 10:57:22,390 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-5) [5967886e] Correlation ID: 5967886e, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-09-10 10:57:22,392 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-1) [3fe2303c] Correlation ID: 3fe2303c, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-09-10 10:57:22,400 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-9) [29632489] Correlation ID: 29632489, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-09-10 10:57:22,415 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-10) [1742c1b1] Correlation ID: 1742c1b1, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-09-10 10:57:22,420 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp-/127.0.0.1:8702-6) [6e85be44] Running command: LogoutUserCommand internal: false. >2014-09-10 10:57:22,434 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-6) [6e85be44] Correlation ID: 6e85be44, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-09-10 10:57:22,499 INFO [org.ovirt.engine.core.utils.servlet.UnsupportedLocaleHelper] (ajp-/127.0.0.1:8702-3) Invalid locale found in configuration: >2014-09-10 10:57:22,499 INFO [org.ovirt.engine.core.utils.servlet.UnsupportedLocaleHelper] (ajp-/127.0.0.1:8702-3) Invalid locale found in configuration: >2014-09-10 10:58:34,566 INFO [org.ovirt.engine.core.utils.servlet.UnsupportedLocaleHelper] (ajp-/127.0.0.1:8702-8) Invalid locale found in configuration: >2014-09-10 10:58:34,567 INFO [org.ovirt.engine.core.utils.servlet.UnsupportedLocaleHelper] (ajp-/127.0.0.1:8702-8) Invalid locale found in configuration: >2014-09-10 11:01:52,170 INFO [org.ovirt.engine.core.bll.aaa.LoginAdminUserCommand] (ajp-/127.0.0.1:8702-10) Running command: LoginAdminUserCommand internal: false. >2014-09-10 11:01:52,184 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-10) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-09-10 11:01:52,296 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp-/127.0.0.1:8702-4) Running command: LoginUserCommand internal: false. >2014-09-10 11:02:29,224 WARN [org.ovirt.engine.core.bll.network.host.SetupNetworksCommand] (ajp-/127.0.0.1:8702-3) [7d081f43] CanDoAction of action SetupNetworks failed. Reasons:VAR__ACTION__SETUP,VAR__TYPE__NETWORKS,VAR__HOST_STATUS__UP_MAINTENANCE_OR_NON_OPERATIONAL,ACTION_TYPE_FAILED_VDS_STATUS_ILLEGAL >2014-09-10 11:02:39,265 INFO [org.ovirt.engine.core.bll.MaintenanceNumberOfVdssCommand] (org.ovirt.thread.pool-6-thread-48) [6c118c1b] Running command: MaintenanceNumberOfVdssCommand internal: false. Entities affected : ID: 7ebdb254-efb6-480b-8c4e-a7dbdc759ed4 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2014-09-10 11:02:39,482 INFO [org.ovirt.engine.core.bll.MaintenanceVdsCommand] (org.ovirt.thread.pool-6-thread-48) [6c118c1b] Running command: MaintenanceVdsCommand internal: true. Entities affected : ID: 7ebdb254-efb6-480b-8c4e-a7dbdc759ed4 Type: VDS >2014-09-10 11:02:39,596 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-6-thread-48) [6c118c1b] START, SetVdsStatusVDSCommand(HostName = rack, HostId = 7ebdb254-efb6-480b-8c4e-a7dbdc759ed4, status=Maintenance, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 4cb58e28 >2014-09-10 11:02:48,482 INFO [org.ovirt.engine.core.bll.MaintenanceNumberOfVdssCommand] (org.ovirt.thread.pool-6-thread-49) [45edf56d] Running command: MaintenanceNumberOfVdssCommand internal: false. Entities affected : ID: 7ebdb254-efb6-480b-8c4e-a7dbdc759ed4 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2014-09-10 11:02:48,687 INFO [org.ovirt.engine.core.bll.MaintenanceVdsCommand] (org.ovirt.thread.pool-6-thread-49) [45edf56d] Running command: MaintenanceVdsCommand internal: true. Entities affected : ID: 7ebdb254-efb6-480b-8c4e-a7dbdc759ed4 Type: VDS >2014-09-10 11:02:48,757 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-6-thread-49) [45edf56d] START, SetVdsStatusVDSCommand(HostName = rack, HostId = 7ebdb254-efb6-480b-8c4e-a7dbdc759ed4, status=Maintenance, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 11707f33 >2014-09-10 11:03:00,603 INFO [org.ovirt.engine.core.bll.MaintenanceNumberOfVdssCommand] (org.ovirt.thread.pool-6-thread-50) [2233ebd5] Running command: MaintenanceNumberOfVdssCommand internal: false. Entities affected : ID: 7ebdb254-efb6-480b-8c4e-a7dbdc759ed4 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2014-09-10 11:03:00,743 INFO [org.ovirt.engine.core.bll.MaintenanceVdsCommand] (org.ovirt.thread.pool-6-thread-50) [2233ebd5] Running command: MaintenanceVdsCommand internal: true. Entities affected : ID: 7ebdb254-efb6-480b-8c4e-a7dbdc759ed4 Type: VDS >2014-09-10 11:03:00,755 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-6-thread-50) [2233ebd5] START, SetVdsStatusVDSCommand(HostName = rack, HostId = 7ebdb254-efb6-480b-8c4e-a7dbdc759ed4, status=Maintenance, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 387e1109 >2014-09-10 11:03:12,810 INFO [org.ovirt.engine.core.bll.MaintenanceNumberOfVdssCommand] (org.ovirt.thread.pool-6-thread-1) [71ce14f4] Running command: MaintenanceNumberOfVdssCommand internal: false. Entities affected : ID: 7ebdb254-efb6-480b-8c4e-a7dbdc759ed4 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2014-09-10 11:03:12,856 INFO [org.ovirt.engine.core.bll.MaintenanceVdsCommand] (org.ovirt.thread.pool-6-thread-1) [71ce14f4] Running command: MaintenanceVdsCommand internal: true. Entities affected : ID: 7ebdb254-efb6-480b-8c4e-a7dbdc759ed4 Type: VDS >2014-09-10 11:03:12,890 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-6-thread-1) [71ce14f4] START, SetVdsStatusVDSCommand(HostName = rack, HostId = 7ebdb254-efb6-480b-8c4e-a7dbdc759ed4, status=Maintenance, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: fa65bc0 >2014-09-10 11:03:46,629 INFO [org.ovirt.engine.core.bll.MaintenanceNumberOfVdssCommand] (org.ovirt.thread.pool-6-thread-2) [58600e60] Running command: MaintenanceNumberOfVdssCommand internal: false. Entities affected : ID: 7ebdb254-efb6-480b-8c4e-a7dbdc759ed4 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2014-09-10 11:03:46,675 INFO [org.ovirt.engine.core.bll.MaintenanceVdsCommand] (org.ovirt.thread.pool-6-thread-2) [58600e60] Running command: MaintenanceVdsCommand internal: true. Entities affected : ID: 7ebdb254-efb6-480b-8c4e-a7dbdc759ed4 Type: VDS >2014-09-10 11:03:46,680 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-6-thread-2) [58600e60] START, SetVdsStatusVDSCommand(HostName = rack, HostId = 7ebdb254-efb6-480b-8c4e-a7dbdc759ed4, status=Maintenance, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 75a14fb >2014-09-10 11:04:08,533 INFO [org.ovirt.engine.core.bll.storage.FenceVdsManualyCommand] (ajp-/127.0.0.1:8702-2) [3d5e1254] Lock Acquired to object EngineLock [exclusiveLocks= key: 7ebdb254-efb6-480b-8c4e-a7dbdc759ed4 value: VDS_FENCE >, sharedLocks= ] >2014-09-10 11:04:08,543 INFO [org.ovirt.engine.core.bll.storage.FenceVdsManualyCommand] (org.ovirt.thread.pool-6-thread-3) [3d5e1254] Running command: FenceVdsManualyCommand internal: false. Entities affected : ID: 7ebdb254-efb6-480b-8c4e-a7dbdc759ed4 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2014-09-10 11:04:08,555 INFO [org.ovirt.engine.core.bll.ClearNonResponsiveVdsVmsCommand] (org.ovirt.thread.pool-6-thread-3) [39e2f8ca] Running command: ClearNonResponsiveVdsVmsCommand internal: true. Entities affected : ID: 7ebdb254-efb6-480b-8c4e-a7dbdc759ed4 Type: VDS >2014-09-10 11:04:08,564 INFO [org.ovirt.engine.core.vdsbroker.UpdateVdsVMsClearedVDSCommand] (org.ovirt.thread.pool-6-thread-3) [39e2f8ca] START, UpdateVdsVMsClearedVDSCommand(HostName = rack, HostId = 7ebdb254-efb6-480b-8c4e-a7dbdc759ed4), log id: 30e21ebf >2014-09-10 11:04:17,298 INFO [org.ovirt.engine.core.bll.MaintenanceNumberOfVdssCommand] (org.ovirt.thread.pool-6-thread-4) [77f83a9a] Running command: MaintenanceNumberOfVdssCommand internal: false. Entities affected : ID: 7ebdb254-efb6-480b-8c4e-a7dbdc759ed4 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2014-09-10 11:04:17,349 INFO [org.ovirt.engine.core.bll.MaintenanceVdsCommand] (org.ovirt.thread.pool-6-thread-4) [77f83a9a] Running command: MaintenanceVdsCommand internal: true. Entities affected : ID: 7ebdb254-efb6-480b-8c4e-a7dbdc759ed4 Type: VDS >2014-09-10 11:04:17,354 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-6-thread-4) [77f83a9a] START, SetVdsStatusVDSCommand(HostName = rack, HostId = 7ebdb254-efb6-480b-8c4e-a7dbdc759ed4, status=Maintenance, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 7725d9ff >2014-09-10 11:04:43,769 INFO [org.ovirt.engine.core.bll.storage.FenceVdsManualyCommand] (ajp-/127.0.0.1:8702-5) [44f99934] Failed to Acquire Lock to object EngineLock [exclusiveLocks= key: 7ebdb254-efb6-480b-8c4e-a7dbdc759ed4 value: VDS_FENCE >, sharedLocks= ] >2014-09-10 11:04:43,770 WARN [org.ovirt.engine.core.bll.storage.FenceVdsManualyCommand] (ajp-/127.0.0.1:8702-5) [44f99934] CanDoAction of action FenceVdsManualy failed. Reasons:VAR__ACTION__MANUAL_FENCE,VAR__TYPE__HOST,POWER_MANAGEMENT_ACTION_ON_ENTITY_ALREADY_IN_PROGRESS >2014-09-10 11:14:08,552 ERROR [org.ovirt.engine.core.bll.storage.FenceVdsManualyCommand] (Transaction Reaper Worker 0) Transaction rolled-back for command: org.ovirt.engine.core.bll.storage.FenceVdsManualyCommand. >2014-09-10 11:47:52,538 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-90) [18a5c631] Principal admin::internal synchronized >2014-09-10 11:47:52,668 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-99) Attempting to update VMs/Templates Ovf. >2014-09-10 12:21:59,203 INFO [org.ovirt.engine.core.utils.servlet.UnsupportedLocaleHelper] (ajp-/127.0.0.1:8702-11) Invalid locale found in configuration: >2014-09-10 12:21:59,203 INFO [org.ovirt.engine.core.utils.servlet.UnsupportedLocaleHelper] (ajp-/127.0.0.1:8702-11) Invalid locale found in configuration: >2014-09-10 12:26:22,938 INFO [org.ovirt.engine.core.bll.MaintenanceNumberOfVdssCommand] (org.ovirt.thread.pool-6-thread-6) [c82eb57] Running command: MaintenanceNumberOfVdssCommand internal: false. Entities affected : ID: 7ebdb254-efb6-480b-8c4e-a7dbdc759ed4 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2014-09-10 12:26:23,039 INFO [org.ovirt.engine.core.bll.MaintenanceVdsCommand] (org.ovirt.thread.pool-6-thread-6) [c82eb57] Running command: MaintenanceVdsCommand internal: true. Entities affected : ID: 7ebdb254-efb6-480b-8c4e-a7dbdc759ed4 Type: VDS >2014-09-10 12:26:23,046 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-6-thread-6) [c82eb57] START, SetVdsStatusVDSCommand(HostName = rack, HostId = 7ebdb254-efb6-480b-8c4e-a7dbdc759ed4, status=Maintenance, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 52ce6a41 >2014-09-10 12:26:41,951 ERROR [org.ovirt.engine.core.utils.timer.SchedulerUtilQuartzImpl] (DefaultQuartzScheduler_Worker-58) Failed to invoke scheduled method refreshLightWeightData: java.lang.reflect.InvocationTargetException > at sun.reflect.GeneratedMethodAccessor41.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: org.springframework.jdbc.CannotGetJdbcConnectionException: Could not get JDBC Connection; nested exception is java.sql.SQLException: javax.resource.ResourceException: IJ000451: The connection manager is shutdown: java:/ENGINEDataSource > at org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:80) [spring-jdbc.jar:3.1.1.RELEASE] > at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:573) [spring-jdbc.jar:3.1.1.RELEASE] > at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:637) [spring-jdbc.jar:3.1.1.RELEASE] > at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:666) [spring-jdbc.jar:3.1.1.RELEASE] > at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:706) [spring-jdbc.jar:3.1.1.RELEASE] > at org.ovirt.engine.core.dal.dbbroker.PostgresDbEngineDialect$PostgresSimpleJdbcCall.executeCallInternal(PostgresDbEngineDialect.java:154) [dal.jar:] > at org.ovirt.engine.core.dal.dbbroker.PostgresDbEngineDialect$PostgresSimpleJdbcCall.doExecute(PostgresDbEngineDialect.java:120) [dal.jar:] > at org.springframework.jdbc.core.simple.SimpleJdbcCall.execute(SimpleJdbcCall.java:181) [spring-jdbc.jar:3.1.1.RELEASE] > at org.ovirt.engine.core.dal.dbbroker.SimpleJdbcCallsHandler.executeImpl(SimpleJdbcCallsHandler.java:141) [dal.jar:] > at org.ovirt.engine.core.dal.dbbroker.SimpleJdbcCallsHandler.executeReadList(SimpleJdbcCallsHandler.java:103) [dal.jar:] > at org.ovirt.engine.core.dao.VdsGroupDAODbFacadeImpl.getAll(VdsGroupDAODbFacadeImpl.java:135) [dal.jar:] > at org.ovirt.engine.core.dao.VdsGroupDAODbFacadeImpl.getAll(VdsGroupDAODbFacadeImpl.java:128) [dal.jar:] > at org.ovirt.engine.core.bll.gluster.GlusterSyncJob.refreshLightWeightData(GlusterSyncJob.java:82) [bll.jar:] > ... 6 more >Caused by: java.sql.SQLException: javax.resource.ResourceException: IJ000451: The connection manager is shutdown: java:/ENGINEDataSource > at org.jboss.jca.adapters.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:151) > at org.springframework.jdbc.datasource.DataSourceUtils.doGetConnection(DataSourceUtils.java:111) [spring-jdbc.jar:3.1.1.RELEASE] > at org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:77) [spring-jdbc.jar:3.1.1.RELEASE] > ... 18 more >Caused by: javax.resource.ResourceException: IJ000451: The connection manager is shutdown: java:/ENGINEDataSource > at org.jboss.jca.core.connectionmanager.AbstractConnectionManager.getManagedConnection(AbstractConnectionManager.java:345) > at org.jboss.jca.core.connectionmanager.tx.TxConnectionManagerImpl.getManagedConnection(TxConnectionManagerImpl.java:368) > at org.jboss.jca.core.connectionmanager.AbstractConnectionManager.allocateConnection(AbstractConnectionManager.java:488) > at org.jboss.jca.adapters.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:143) > ... 20 more > >2014-09-10 12:26:41,997 ERROR [org.ovirt.engine.core.utils.timer.SchedulerUtilQuartzImpl] (DefaultQuartzScheduler_Worker-68) Failed to invoke scheduled method refreshGlusterServices: java.lang.reflect.InvocationTargetException > at sun.reflect.GeneratedMethodAccessor40.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: org.springframework.jdbc.CannotGetJdbcConnectionException: Could not get JDBC Connection; nested exception is java.sql.SQLException: javax.resource.ResourceException: IJ000451: The connection manager is shutdown: java:/ENGINEDataSource > at org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:80) [spring-jdbc.jar:3.1.1.RELEASE] > at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:573) [spring-jdbc.jar:3.1.1.RELEASE] > at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:637) [spring-jdbc.jar:3.1.1.RELEASE] > at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:666) [spring-jdbc.jar:3.1.1.RELEASE] > at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:706) [spring-jdbc.jar:3.1.1.RELEASE] > at org.ovirt.engine.core.dal.dbbroker.PostgresDbEngineDialect$PostgresSimpleJdbcCall.executeCallInternal(PostgresDbEngineDialect.java:154) [dal.jar:] > at org.ovirt.engine.core.dal.dbbroker.PostgresDbEngineDialect$PostgresSimpleJdbcCall.doExecute(PostgresDbEngineDialect.java:120) [dal.jar:] > at org.springframework.jdbc.core.simple.SimpleJdbcCall.execute(SimpleJdbcCall.java:181) [spring-jdbc.jar:3.1.1.RELEASE] > at org.ovirt.engine.core.dal.dbbroker.SimpleJdbcCallsHandler.executeImpl(SimpleJdbcCallsHandler.java:141) [dal.jar:] > at org.ovirt.engine.core.dal.dbbroker.SimpleJdbcCallsHandler.executeReadList(SimpleJdbcCallsHandler.java:103) [dal.jar:] > at org.ovirt.engine.core.dao.VdsGroupDAODbFacadeImpl.getAll(VdsGroupDAODbFacadeImpl.java:135) [dal.jar:] > at org.ovirt.engine.core.dao.VdsGroupDAODbFacadeImpl.getAll(VdsGroupDAODbFacadeImpl.java:128) [dal.jar:] > at org.ovirt.engine.core.bll.gluster.GlusterServiceSyncJob.refreshGlusterServices(GlusterServiceSyncJob.java:54) [bll.jar:] > ... 6 more >Caused by: java.sql.SQLException: javax.resource.ResourceException: IJ000451: The connection manager is shutdown: java:/ENGINEDataSource > at org.jboss.jca.adapters.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:151) > at org.springframework.jdbc.datasource.DataSourceUtils.doGetConnection(DataSourceUtils.java:111) [spring-jdbc.jar:3.1.1.RELEASE] > at org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:77) [spring-jdbc.jar:3.1.1.RELEASE] > ... 18 more >Caused by: javax.resource.ResourceException: IJ000451: The connection manager is shutdown: java:/ENGINEDataSource > at org.jboss.jca.core.connectionmanager.AbstractConnectionManager.getManagedConnection(AbstractConnectionManager.java:345) > at org.jboss.jca.core.connectionmanager.tx.TxConnectionManagerImpl.getManagedConnection(TxConnectionManagerImpl.java:368) > at org.jboss.jca.core.connectionmanager.AbstractConnectionManager.allocateConnection(AbstractConnectionManager.java:488) > at org.jboss.jca.adapters.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:143) > ... 20 more > >2014-09-10 12:26:46,958 ERROR [org.ovirt.engine.core.utils.timer.SchedulerUtilQuartzImpl] (DefaultQuartzScheduler_Worker-52) Failed to invoke scheduled method refreshLightWeightData: java.lang.reflect.InvocationTargetException > at sun.reflect.GeneratedMethodAccessor41.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: org.springframework.jdbc.CannotGetJdbcConnectionException: Could not get JDBC Connection; nested exception is java.sql.SQLException: javax.resource.ResourceException: IJ000451: The connection manager is shutdown: java:/ENGINEDataSource > at org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:80) [spring-jdbc.jar:3.1.1.RELEASE] > at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:573) [spring-jdbc.jar:3.1.1.RELEASE] > at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:637) [spring-jdbc.jar:3.1.1.RELEASE] > at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:666) [spring-jdbc.jar:3.1.1.RELEASE] > at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:706) [spring-jdbc.jar:3.1.1.RELEASE] > at org.ovirt.engine.core.dal.dbbroker.PostgresDbEngineDialect$PostgresSimpleJdbcCall.executeCallInternal(PostgresDbEngineDialect.java:154) [dal.jar:] > at org.ovirt.engine.core.dal.dbbroker.PostgresDbEngineDialect$PostgresSimpleJdbcCall.doExecute(PostgresDbEngineDialect.java:120) [dal.jar:] > at org.springframework.jdbc.core.simple.SimpleJdbcCall.execute(SimpleJdbcCall.java:181) [spring-jdbc.jar:3.1.1.RELEASE] > at org.ovirt.engine.core.dal.dbbroker.SimpleJdbcCallsHandler.executeImpl(SimpleJdbcCallsHandler.java:141) [dal.jar:] > at org.ovirt.engine.core.dal.dbbroker.SimpleJdbcCallsHandler.executeReadList(SimpleJdbcCallsHandler.java:103) [dal.jar:] > at org.ovirt.engine.core.dao.VdsGroupDAODbFacadeImpl.getAll(VdsGroupDAODbFacadeImpl.java:135) [dal.jar:] > at org.ovirt.engine.core.dao.VdsGroupDAODbFacadeImpl.getAll(VdsGroupDAODbFacadeImpl.java:128) [dal.jar:] > at org.ovirt.engine.core.bll.gluster.GlusterSyncJob.refreshLightWeightData(GlusterSyncJob.java:82) [bll.jar:] > ... 6 more >Caused by: java.sql.SQLException: javax.resource.ResourceException: IJ000451: The connection manager is shutdown: java:/ENGINEDataSource > at org.jboss.jca.adapters.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:151) > at org.springframework.jdbc.datasource.DataSourceUtils.doGetConnection(DataSourceUtils.java:111) [spring-jdbc.jar:3.1.1.RELEASE] > at org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:77) [spring-jdbc.jar:3.1.1.RELEASE] > ... 18 more >Caused by: javax.resource.ResourceException: IJ000451: The connection manager is shutdown: java:/ENGINEDataSource > at org.jboss.jca.core.connectionmanager.AbstractConnectionManager.getManagedConnection(AbstractConnectionManager.java:345) > at org.jboss.jca.core.connectionmanager.tx.TxConnectionManagerImpl.getManagedConnection(TxConnectionManagerImpl.java:368) > at org.jboss.jca.core.connectionmanager.AbstractConnectionManager.allocateConnection(AbstractConnectionManager.java:488) > at org.jboss.jca.adapters.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:143) > ... 20 more > >2014-09-10 12:26:47,003 ERROR [org.ovirt.engine.core.utils.timer.SchedulerUtilQuartzImpl] (DefaultQuartzScheduler_Worker-76) Failed to invoke scheduled method refreshGlusterServices: java.lang.reflect.InvocationTargetException > at sun.reflect.GeneratedMethodAccessor40.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: org.springframework.jdbc.CannotGetJdbcConnectionException: Could not get JDBC Connection; nested exception is java.sql.SQLException: javax.resource.ResourceException: IJ000451: The connection manager is shutdown: java:/ENGINEDataSource > at org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:80) [spring-jdbc.jar:3.1.1.RELEASE] > at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:573) [spring-jdbc.jar:3.1.1.RELEASE] > at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:637) [spring-jdbc.jar:3.1.1.RELEASE] > at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:666) [spring-jdbc.jar:3.1.1.RELEASE] > at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:706) [spring-jdbc.jar:3.1.1.RELEASE] > at org.ovirt.engine.core.dal.dbbroker.PostgresDbEngineDialect$PostgresSimpleJdbcCall.executeCallInternal(PostgresDbEngineDialect.java:154) [dal.jar:] > at org.ovirt.engine.core.dal.dbbroker.PostgresDbEngineDialect$PostgresSimpleJdbcCall.doExecute(PostgresDbEngineDialect.java:120) [dal.jar:] > at org.springframework.jdbc.core.simple.SimpleJdbcCall.execute(SimpleJdbcCall.java:181) [spring-jdbc.jar:3.1.1.RELEASE] > at org.ovirt.engine.core.dal.dbbroker.SimpleJdbcCallsHandler.executeImpl(SimpleJdbcCallsHandler.java:141) [dal.jar:] > at org.ovirt.engine.core.dal.dbbroker.SimpleJdbcCallsHandler.executeReadList(SimpleJdbcCallsHandler.java:103) [dal.jar:] > at org.ovirt.engine.core.dao.VdsGroupDAODbFacadeImpl.getAll(VdsGroupDAODbFacadeImpl.java:135) [dal.jar:] > at org.ovirt.engine.core.dao.VdsGroupDAODbFacadeImpl.getAll(VdsGroupDAODbFacadeImpl.java:128) [dal.jar:] > at org.ovirt.engine.core.bll.gluster.GlusterServiceSyncJob.refreshGlusterServices(GlusterServiceSyncJob.java:54) [bll.jar:] > ... 6 more >Caused by: java.sql.SQLException: javax.resource.ResourceException: IJ000451: The connection manager is shutdown: java:/ENGINEDataSource > at org.jboss.jca.adapters.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:151) > at org.springframework.jdbc.datasource.DataSourceUtils.doGetConnection(DataSourceUtils.java:111) [spring-jdbc.jar:3.1.1.RELEASE] > at org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:77) [spring-jdbc.jar:3.1.1.RELEASE] > ... 18 more >Caused by: javax.resource.ResourceException: IJ000451: The connection manager is shutdown: java:/ENGINEDataSource > at org.jboss.jca.core.connectionmanager.AbstractConnectionManager.getManagedConnection(AbstractConnectionManager.java:345) > at org.jboss.jca.core.connectionmanager.tx.TxConnectionManagerImpl.getManagedConnection(TxConnectionManagerImpl.java:368) > at org.jboss.jca.core.connectionmanager.AbstractConnectionManager.allocateConnection(AbstractConnectionManager.java:488) > at org.jboss.jca.adapters.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:143) > ... 20 more > >2014-09-10 12:26:58,780 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Loaded file "/usr/share/ovirt-engine/services/ovirt-engine/ovirt-engine.conf". >2014-09-10 12:26:58,781 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) The file "/etc/ovirt-engine/engine.conf" doesn't exist or isn't readable. Will return an empty set of properties. >2014-09-10 12:26:58,783 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Loaded file "/etc/ovirt-engine/engine.conf.d/10-rhevm-doc.conf". >2014-09-10 12:26:58,785 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Loaded file "/etc/ovirt-engine/engine.conf.d/10-setup-database.conf". >2014-09-10 12:26:58,786 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Loaded file "/etc/ovirt-engine/engine.conf.d/10-setup-jboss.conf". >2014-09-10 12:26:58,795 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Loaded file "/etc/ovirt-engine/engine.conf.d/10-setup-pki.conf". >2014-09-10 12:26:58,796 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Loaded file "/etc/ovirt-engine/engine.conf.d/10-setup-protocols.conf". >2014-09-10 12:26:58,796 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Loaded file "/etc/ovirt-engine/engine.conf.d/20-branding-rhev.conf". >2014-09-10 12:26:58,796 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Loaded file "/etc/ovirt-engine/engine.conf.d/50-redhat-support-plugin.conf". >2014-09-10 12:26:58,797 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "ENGINE_AJP_ENABLED" is "true". >2014-09-10 12:26:58,797 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "ENGINE_AJP_PORT" is "8702". >2014-09-10 12:26:58,797 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "ENGINE_APPS" is "engine.ear restapi.war legacy_restapi.war "/usr/share/ovirt-engine/branding/rhev-2.brand/applications/rhev.ear" redhat-support-plugin-rhev.ear". >2014-09-10 12:26:58,798 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "ENGINE_CACHE" is "/var/cache/ovirt-engine". >2014-09-10 12:26:58,798 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "ENGINE_DB_CHECK_INTERVAL" is "1000". >2014-09-10 12:26:58,798 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "ENGINE_DB_CONNECTION_TIMEOUT" is "300000". >2014-09-10 12:26:58,798 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "ENGINE_DB_DATABASE" is "engine". >2014-09-10 12:26:58,798 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "ENGINE_DB_DRIVER" is "org.postgresql.Driver". >2014-09-10 12:26:58,798 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "ENGINE_DB_HOST" is "localhost". >2014-09-10 12:26:58,799 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "ENGINE_DB_MAX_CONNECTIONS" is "100". >2014-09-10 12:26:58,799 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "ENGINE_DB_MIN_CONNECTIONS" is "1". >2014-09-10 12:26:58,799 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "ENGINE_DB_PASSWORD" is "***". >2014-09-10 12:26:58,799 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "ENGINE_DB_PORT" is "5432". >2014-09-10 12:26:58,803 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "ENGINE_DB_SECURED" is "False". >2014-09-10 12:26:58,803 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "ENGINE_DB_SECURED_VALIDATION" is "False". >2014-09-10 12:26:58,804 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "ENGINE_DB_URL" is "jdbc:postgresql://localhost:5432/engine?sslfactory=org.postgresql.ssl.NonValidatingFactory". >2014-09-10 12:26:58,804 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "ENGINE_DB_USER" is "engine". >2014-09-10 12:26:58,804 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "ENGINE_DEBUG_ADDRESS" is "". >2014-09-10 12:26:58,804 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "ENGINE_DOC" is "/usr/share/doc/ovirt-engine". >2014-09-10 12:26:58,804 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "ENGINE_ETC" is "/etc/ovirt-engine". >2014-09-10 12:26:58,805 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "ENGINE_EXTENSION_PATH" is "/usr/share/ovirt-engine/extensions.d:/etc/ovirt-engine/extensions.d". >2014-09-10 12:26:58,805 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "ENGINE_FQDN" is "pb-rh35.rhev.lab.eng.brq.redhat.com". >2014-09-10 12:26:58,805 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "ENGINE_GROUP" is "ovirt". >2014-09-10 12:26:58,805 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "ENGINE_HEAP_MAX" is "1g". >2014-09-10 12:26:58,805 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "ENGINE_HEAP_MIN" is "1g". >2014-09-10 12:26:58,806 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "ENGINE_HTTPS_ENABLED" is "false". >2014-09-10 12:26:58,806 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "ENGINE_HTTPS_PORT" is "None". >2014-09-10 12:26:58,806 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "ENGINE_HTTPS_PROTOCOLS" is "SSLv3,TLSv1,TLSv1.1,TLSv1.2". >2014-09-10 12:26:58,806 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "ENGINE_HTTP_ENABLED" is "false". >2014-09-10 12:26:58,806 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "ENGINE_HTTP_PORT" is "None". >2014-09-10 12:26:58,806 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "ENGINE_JAVA_MODULEPATH" is "/usr/share/ovirt-engine/modules". >2014-09-10 12:26:58,807 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "ENGINE_JVM_ARGS" is " -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath="/var/log/ovirt-engine/dump"". >2014-09-10 12:26:58,807 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "ENGINE_LOG" is "/var/log/ovirt-engine". >2014-09-10 12:26:58,807 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "ENGINE_LOG_TO_CONSOLE" is "false". >2014-09-10 12:26:58,815 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "ENGINE_MANUAL" is "/usr/share/doc/rhevm-doc/manual". >2014-09-10 12:26:58,816 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "ENGINE_PERM_MAX" is "256m". >2014-09-10 12:26:58,816 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "ENGINE_PERM_MIN" is "256m". >2014-09-10 12:26:58,816 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "ENGINE_PKI" is "/etc/pki/ovirt-engine". >2014-09-10 12:26:58,816 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "ENGINE_PKI_CA" is "/etc/pki/ovirt-engine/ca.pem". >2014-09-10 12:26:58,816 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "ENGINE_PKI_ENGINE_CERT" is "/etc/pki/ovirt-engine/certs/engine.cer". >2014-09-10 12:26:58,824 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "ENGINE_PKI_ENGINE_STORE" is "/etc/pki/ovirt-engine/keys/engine.p12". >2014-09-10 12:26:58,825 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "ENGINE_PKI_ENGINE_STORE_ALIAS" is "1". >2014-09-10 12:26:58,825 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "ENGINE_PKI_ENGINE_STORE_PASSWORD" is "***". >2014-09-10 12:26:58,825 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "ENGINE_PKI_TRUST_STORE" is "/etc/pki/ovirt-engine/.truststore". >2014-09-10 12:26:58,825 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "ENGINE_PKI_TRUST_STORE_PASSWORD" is "***". >2014-09-10 12:26:58,825 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "ENGINE_PROPERTIES" is " jsse.enableSNIExtension=false "java.security.krb5.conf=/etc/ovirt-engine/krb5.conf"". >2014-09-10 12:26:58,825 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "ENGINE_PROXY_ENABLED" is "true". >2014-09-10 12:26:58,826 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "ENGINE_PROXY_HTTPS_PORT" is "443". >2014-09-10 12:26:58,826 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "ENGINE_PROXY_HTTP_PORT" is "80". >2014-09-10 12:26:58,826 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "ENGINE_REPORTS_UI" is "/var/lib/ovirt-engine/reports.xml". >2014-09-10 12:26:58,826 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "ENGINE_STOP_INTERVAL" is "1". >2014-09-10 12:26:58,826 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "ENGINE_STOP_TIME" is "10". >2014-09-10 12:26:58,826 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "ENGINE_TMP" is "/var/tmp/ovirt-engine". >2014-09-10 12:26:58,826 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "ENGINE_UP_MARK" is "/var/lib/ovirt-engine/engine.up". >2014-09-10 12:26:58,827 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "ENGINE_URI" is "/ovirt-engine". >2014-09-10 12:26:58,827 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "ENGINE_USER" is "ovirt". >2014-09-10 12:26:58,827 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "ENGINE_USR" is "/usr/share/ovirt-engine". >2014-09-10 12:26:58,827 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "ENGINE_VAR" is "/var/lib/ovirt-engine". >2014-09-10 12:26:58,827 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "ENGINE_VERBOSE_GC" is "false". >2014-09-10 12:26:58,827 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "JBOSS_HOME" is "/usr/share/jbossas". >2014-09-10 12:26:58,827 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "JBOSS_RUNTIME" is "/var/lib/ovirt-engine/jboss_runtime". >2014-09-10 12:26:58,828 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 39) Value of property "SENSITIVE_KEYS" is ",ENGINE_DB_PASSWORD,ENGINE_PKI_TRUST_STORE_PASSWORD,ENGINE_PKI_ENGINE_STORE_PASSWORD". >2014-09-10 12:27:09,063 INFO [org.ovirt.engine.core.bll.Backend] (ajp-/127.0.0.1:8702-1) Start initializing Backend >2014-09-10 12:27:09,691 INFO [org.ovirt.engine.core.utils.osinfo.OsInfoPreferencesLoader] (ajp-/127.0.0.1:8702-1) Loaded file /etc/ovirt-engine/osinfo.conf.d/00-defaults.properties >2014-09-10 12:27:09,864 INFO [org.ovirt.engine.core.bll.Backend] (ajp-/127.0.0.1:8702-1) Running ovirt-engine 3.5.0-0.10.master.el6ev >2014-09-10 12:27:09,865 INFO [org.ovirt.engine.core.bll.CpuFlagsManagerHandler] (ajp-/127.0.0.1:8702-1) Start initializing dictionaries >2014-09-10 12:27:09,879 INFO [org.ovirt.engine.core.bll.CpuFlagsManagerHandler] (ajp-/127.0.0.1:8702-1) Finished initializing dictionaries >2014-09-10 12:27:09,880 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (ajp-/127.0.0.1:8702-1) Start initializing AuditLogCleanupManager >2014-09-10 12:27:09,880 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (ajp-/127.0.0.1:8702-1) Setting audit cleanup manager to run at: 35 35 3 * * ? >2014-09-10 12:27:09,918 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (ajp-/127.0.0.1:8702-1) Finished initializing AuditLogCleanupManager >2014-09-10 12:27:09,918 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (ajp-/127.0.0.1:8702-1) Start initializing CommandEntityCleanupManager >2014-09-10 12:27:09,919 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (ajp-/127.0.0.1:8702-1) Setting command entity cleanup manager to run at: 35 35 3 * * ? >2014-09-10 12:27:09,921 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (ajp-/127.0.0.1:8702-1) Finished initializing CommandEntityCleanupManager >2014-09-10 12:27:09,925 INFO [org.ovirt.engine.core.bll.TagsDirector] (ajp-/127.0.0.1:8702-1) Start initializing TagsDirector >2014-09-10 12:27:09,963 INFO [org.ovirt.engine.core.bll.TagsDirector] (ajp-/127.0.0.1:8702-1) Tag root added to tree >2014-09-10 12:27:09,985 INFO [org.ovirt.engine.core.bll.TagsDirector] (ajp-/127.0.0.1:8702-1) Finished initializing TagsDirector >2014-09-10 12:27:09,986 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (ajp-/127.0.0.1:8702-1) Start initializing IsoDomainListSyncronizer >2014-09-10 12:27:09,998 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (ajp-/127.0.0.1:8702-1) Finished initializing IsoDomainListSyncronizer >2014-09-10 12:27:10,074 INFO [org.ovirt.engine.core.bll.Backend] (ajp-/127.0.0.1:8702-1) Completed initializing handlers >2014-09-10 12:27:10,124 INFO [org.ovirt.engine.core.utils.ErrorTranslatorImpl] (ajp-/127.0.0.1:8702-1) Start initializing ErrorTranslatorImpl >2014-09-10 12:27:10,154 WARN [org.ovirt.engine.core.utils.ErrorTranslatorImpl] (ajp-/127.0.0.1:8702-1) Code MAC_ADDRESS_IS_IN_USE appears more than once in string table. >2014-09-10 12:27:10,156 INFO [org.ovirt.engine.core.utils.ErrorTranslatorImpl] (ajp-/127.0.0.1:8702-1) Finished initializing ErrorTranslatorImpl >2014-09-10 12:27:10,156 INFO [org.ovirt.engine.core.utils.ErrorTranslatorImpl] (ajp-/127.0.0.1:8702-1) Start initializing ErrorTranslatorImpl >2014-09-10 12:27:10,158 INFO [org.ovirt.engine.core.utils.ErrorTranslatorImpl] (ajp-/127.0.0.1:8702-1) Finished initializing ErrorTranslatorImpl >2014-09-10 12:27:10,158 INFO [org.ovirt.engine.core.bll.Backend] (ajp-/127.0.0.1:8702-1) Mark incomplete jobs as UNKNOWN >2014-09-10 12:27:10,231 INFO [org.ovirt.engine.core.bll.job.JobRepositoryCleanupManager] (ajp-/127.0.0.1:8702-1) Start initializing JobRepositoryCleanupManager >2014-09-10 12:27:10,232 INFO [org.ovirt.engine.core.bll.job.JobRepositoryCleanupManager] (ajp-/127.0.0.1:8702-1) Finished initializing JobRepositoryCleanupManager >2014-09-10 12:27:10,232 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (ajp-/127.0.0.1:8702-1) Start initializing AutoRecoveryManager >2014-09-10 12:27:10,234 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (ajp-/127.0.0.1:8702-1) Finished initializing AutoRecoveryManager >2014-09-10 12:27:10,237 INFO [org.ovirt.engine.core.dal.job.ExecutionMessageDirector] (ajp-/127.0.0.1:8702-1) Start initializing ExecutionMessageDirector >2014-09-10 12:27:10,248 INFO [org.ovirt.engine.core.dal.job.ExecutionMessageDirector] (ajp-/127.0.0.1:8702-1) Finished initializing ExecutionMessageDirector >2014-09-10 12:27:10,463 INFO [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (ajp-/127.0.0.1:8702-1) Loading extension 'builtin-authn-internal' >2014-09-10 12:27:10,470 INFO [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (ajp-/127.0.0.1:8702-1) Extension 'builtin-authn-internal' loaded >2014-09-10 12:27:10,473 INFO [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (ajp-/127.0.0.1:8702-1) Loading extension 'internal' >2014-09-10 12:27:10,476 INFO [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (ajp-/127.0.0.1:8702-1) Extension 'internal' loaded >2014-09-10 12:27:10,477 INFO [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (ajp-/127.0.0.1:8702-1) Initializing extension 'builtin-authn-internal' >2014-09-10 12:27:10,477 INFO [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (ajp-/127.0.0.1:8702-1) Extension 'builtin-authn-internal' initialized >2014-09-10 12:27:10,480 INFO [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (ajp-/127.0.0.1:8702-1) Initializing extension 'internal' >2014-09-10 12:27:10,480 INFO [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (ajp-/127.0.0.1:8702-1) Extension 'internal' initialized >2014-09-10 12:27:10,480 INFO [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (ajp-/127.0.0.1:8702-1) Start of enabled extensions list >2014-09-10 12:27:10,481 INFO [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (ajp-/127.0.0.1:8702-1) Instance name: 'builtin-authn-internal', Extension name: 'Internal Authn (Built-in)', Version: 'N/A', Notes: '', License: 'ASL 2.0', Home: 'http://www.ovirt.org', Author 'The oVirt Project', Build interface Version: '0', File: 'N/A', Initialized: 'true' >2014-09-10 12:27:10,481 INFO [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (ajp-/127.0.0.1:8702-1) Instance name: 'internal', Extension name: 'Internal Authz (Built-in)', Version: 'N/A', Notes: '', License: 'ASL 2.0', Home: 'http://www.ovirt.org', Author 'The oVirt Project', Build interface Version: '0', File: 'N/A', Initialized: 'true' >2014-09-10 12:27:10,481 INFO [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (ajp-/127.0.0.1:8702-1) End of enabled extensions list >2014-09-10 12:27:10,843 INFO [org.ovirt.engine.ui.frontend.server.gwt.plugin.PluginDataManager] (ajp-/127.0.0.1:8702-1) Reading UI plugin descriptor [/usr/share/ovirt-engine/ui-plugins/redhat_support_plugin_rhev.json] >2014-09-10 12:27:10,878 INFO [org.ovirt.engine.core.bll.aaa.DbUserCacheManager] (ServerService Thread Pool -- 49) Start initializing DbUserCacheManager >2014-09-10 12:27:10,881 INFO [org.ovirt.engine.ui.frontend.server.gwt.plugin.PluginDataManager] (ajp-/127.0.0.1:8702-1) Reading UI plugin configuration [/etc/ovirt-engine/ui-plugins/redhat_support_plugin_rhev-config.json] >2014-09-10 12:27:10,890 INFO [org.ovirt.engine.core.bll.aaa.DbUserCacheManager] (ServerService Thread Pool -- 49) Finished initializing DbUserCacheManager >2014-09-10 12:27:10,950 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (ServerService Thread Pool -- 49) Initialization of AsyncTaskManager completed successfully. >2014-09-10 12:27:10,963 INFO [org.ovirt.engine.core.vdsbroker.ResourceManager] (ServerService Thread Pool -- 49) Start initializing ResourceManager >2014-09-10 12:27:11,143 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-1) Principal admin::internal synchronized >2014-09-10 12:27:11,311 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (ServerService Thread Pool -- 49) Entered VdsManager constructor >2014-09-10 12:27:11,361 INFO [org.ovirt.engine.core.utils.servlet.UnsupportedLocaleHelper] (ajp-/127.0.0.1:8702-1) Invalid locale found in configuration: >2014-09-10 12:27:11,361 INFO [org.ovirt.engine.core.utils.servlet.UnsupportedLocaleHelper] (ajp-/127.0.0.1:8702-1) Invalid locale found in configuration: >2014-09-10 12:27:11,455 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (ServerService Thread Pool -- 49) Initialize vdsBroker (dell-r210ii-16.rhev.lab.eng.brq.redhat.com,54,321) >2014-09-10 12:27:11,804 INFO [org.ovirt.engine.core.vdsbroker.ResourceManager] (ServerService Thread Pool -- 49) VDS 7ebdb254-efb6-480b-8c4e-a7dbdc759ed4 was added to the Resource Manager >2014-09-10 12:27:11,861 INFO [org.ovirt.engine.core.vdsbroker.ResourceManager] (ServerService Thread Pool -- 49) Finished initializing ResourceManager >2014-09-10 12:27:11,869 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (ServerService Thread Pool -- 49) Initialization of OvfDataUpdater completed successfully. >2014-09-10 12:27:11,869 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (ServerService Thread Pool -- 49) Start scheduling to enable vds load balancer >2014-09-10 12:27:11,877 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (ServerService Thread Pool -- 49) Finished scheduling to enable vds load balancer >2014-09-10 12:27:11,877 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (ServerService Thread Pool -- 49) Start HA Reservation check >2014-09-10 12:27:11,878 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (ServerService Thread Pool -- 49) Finished HA Reservation check >2014-09-10 12:27:11,889 INFO [org.ovirt.engine.core.bll.network.MacPoolManagerRanges] (org.ovirt.thread.pool-6-thread-1) Start initializing MacPoolManagerRanges >2014-09-10 12:27:11,907 INFO [org.ovirt.engine.core.bll.InitBackendServicesOnStartupBean] (ServerService Thread Pool -- 49) Init VM custom properties utilities >2014-09-10 12:27:11,907 INFO [org.ovirt.engine.core.bll.InitBackendServicesOnStartupBean] (ServerService Thread Pool -- 49) Init device custom properties utilities >2014-09-10 12:27:11,919 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (ServerService Thread Pool -- 49) Initializing Scheduling manager >2014-09-10 12:27:11,925 INFO [org.ovirt.engine.core.bll.network.MacPoolManagerRanges] (org.ovirt.thread.pool-6-thread-1) Finished initializing. Available MACs in pool: 65 >2014-09-10 12:27:12,168 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (ServerService Thread Pool -- 49) External scheduler disabled, discovery skipped >2014-09-10 12:27:12,168 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (ServerService Thread Pool -- 49) Initialized Scheduling manager >2014-09-10 12:27:12,169 INFO [org.ovirt.engine.core.bll.dwh.DwhHeartBeat] (ServerService Thread Pool -- 49) Initializing DWH Heart Beat >2014-09-10 12:27:12,172 INFO [org.ovirt.engine.core.bll.dwh.DwhHeartBeat] (ServerService Thread Pool -- 49) DWH Heart Beat initialized >2014-09-10 12:27:13,767 INFO [org.ovirt.engine.core.utils.servlet.UnsupportedLocaleHelper] (ajp-/127.0.0.1:8702-2) Invalid locale found in configuration: >2014-09-10 12:27:13,767 INFO [org.ovirt.engine.core.utils.servlet.UnsupportedLocaleHelper] (ajp-/127.0.0.1:8702-2) Invalid locale found in configuration: >2014-09-10 12:27:15,778 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to dell-r210ii-16.rhev.lab.eng.brq.redhat.com/10.34.62.208 >2014-09-10 12:27:15,784 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-10 12:27:15,786 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-10 12:27:15,789 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-7) Exception during connection: java.util.concurrent.ExecutionException: java.net.ConnectException: Connection refused > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:98) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:56) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:57) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:175) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_65] > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-10 12:27:15,836 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-7) Command GetCapabilitiesVDSCommand(HostName = rack, HostId = 7ebdb254-efb6-480b-8c4e-a7dbdc759ed4, vds=Host[rack,7ebdb254-efb6-480b-8c4e-a7dbdc759ed4]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-09-10 12:27:15,837 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-7) Host rack is not responding. It will stay in Connecting state for a grace period of 60 seconds and after that an attempt to fence the host will be issued. >2014-09-10 12:27:15,943 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-7) AuditLogType: UNASSIGNED not exist in string table >2014-09-10 12:27:15,944 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-7) AuditLogType: VDS_AUTO_FENCE_STATUS not exist in string table >2014-09-10 12:27:15,944 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-7) AuditLogType: VDS_AUTO_FENCE_STATUS_FAILED not exist in string table >2014-09-10 12:27:15,944 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-7) AuditLogType: VDS_AUTO_FENCE_FAILED_CALL_FENCE_SPM not exist in string table >2014-09-10 12:27:15,944 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-7) AuditLogType: VDS_HIGH_NETWORK_USE not exist in string table >2014-09-10 12:27:15,944 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-7) AuditLogType: USER_FAILED_REMOVE_VM not exist in string table >2014-09-10 12:27:15,944 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-7) AuditLogType: USER_RUN_UNLOCK_ENTITY_SCRIPT not exist in string table >2014-09-10 12:27:15,946 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-7) AuditLogType: VDS_NETWORK_MTU_DIFFER_FROM_LOGICAL_NETWORK not exist in string table >2014-09-10 12:27:15,946 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-7) AuditLogType: STORAGE_ACTIVATE_ASYNC not exist in string table >2014-09-10 12:27:15,946 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-7) AuditLogType: DWH_STOPPED not exist in string table >2014-09-10 12:27:15,946 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-7) AuditLogType: DWH_STARTED not exist in string table >2014-09-10 12:27:15,947 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-7) AuditLogType: DWH_ERROR not exist in string table >2014-09-10 12:27:15,947 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-7) AuditLogType: USER_ADDED_DISK_PROFILE not exist in string table >2014-09-10 12:27:15,948 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-7) AuditLogType: USER_FAILED_TO_ADD_DISK_PROFILE not exist in string table >2014-09-10 12:27:15,949 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-7) AuditLogType: USER_REMOVED_DISK_PROFILE not exist in string table >2014-09-10 12:27:15,949 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-7) AuditLogType: USER_FAILED_TO_REMOVE_DISK_PROFILE not exist in string table >2014-09-10 12:27:15,950 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-7) AuditLogType: USER_UPDATED_DISK_PROFILE not exist in string table >2014-09-10 12:27:15,950 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-7) AuditLogType: USER_FAILED_TO_UPDATE_DISK_PROFILE not exist in string table >2014-09-10 12:27:15,950 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-7) AuditLogType: USER_ADDED_CPU_PROFILE not exist in string table >2014-09-10 12:27:15,950 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-7) AuditLogType: USER_FAILED_TO_ADD_CPU_PROFILE not exist in string table >2014-09-10 12:27:15,950 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-7) AuditLogType: USER_REMOVED_CPU_PROFILE not exist in string table >2014-09-10 12:27:15,950 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-7) AuditLogType: USER_FAILED_TO_REMOVE_CPU_PROFILE not exist in string table >2014-09-10 12:27:15,950 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-7) AuditLogType: USER_UPDATED_CPU_PROFILE not exist in string table >2014-09-10 12:27:15,951 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-7) AuditLogType: USER_FAILED_TO_UPDATE_CPU_PROFILE not exist in string table >2014-09-10 12:27:15,951 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-7) AuditLogType: USER_REMOVE_AUDIT_LOG not exist in string table >2014-09-10 12:27:15,951 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-7) AuditLogType: USER_REMOVE_AUDIT_LOG_FAILED not exist in string table >2014-09-10 12:27:15,951 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-7) AuditLogType: USER_CLEAR_ALL_DISMISSED_AUDIT_LOG not exist in string table >2014-09-10 12:27:15,951 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-7) AuditLogType: USER_CLEAR_ALL_DISMISSED_AUDIT_LOG_FAILED not exist in string table >2014-09-10 12:27:16,492 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-7) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host rack is not responding. It will stay in Connecting state for a grace period of 60 seconds and after that an attempt to fence the host will be issued. >2014-09-10 12:27:16,519 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-7) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_65] > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-10 12:27:16,656 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-7) Failed to refresh VDS , vds = 7ebdb254-efb6-480b-8c4e-a7dbdc759ed4 : rack, VDS Network Error, continuing. >java.net.ConnectException: Connection refused >2014-09-10 12:27:19,732 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to dell-r210ii-16.rhev.lab.eng.brq.redhat.com/10.34.62.208 >2014-09-10 12:27:19,733 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-10 12:27:19,733 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-10 12:27:19,734 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-15) Exception during connection: java.util.concurrent.ExecutionException: java.net.ConnectException: Connection refused > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:98) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:56) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:57) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:175) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_65] > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-10 12:27:19,736 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-15) Command GetCapabilitiesVDSCommand(HostName = rack, HostId = 7ebdb254-efb6-480b-8c4e-a7dbdc759ed4, vds=Host[rack,7ebdb254-efb6-480b-8c4e-a7dbdc759ed4]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-09-10 12:27:19,737 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-15) Host rack is not responding. It will stay in Connecting state for a grace period of 60 seconds and after that an attempt to fence the host will be issued. >2014-09-10 12:27:19,765 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-15) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_65] > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-10 12:27:19,783 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-15) Failed to refresh VDS , vds = 7ebdb254-efb6-480b-8c4e-a7dbdc759ed4 : rack, VDS Network Error, continuing. >java.net.ConnectException: Connection refused >2014-09-10 12:27:22,800 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to dell-r210ii-16.rhev.lab.eng.brq.redhat.com/10.34.62.208 >2014-09-10 12:27:22,801 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-10 12:27:22,802 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-10 12:27:22,802 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-24) Exception during connection: java.util.concurrent.ExecutionException: java.net.ConnectException: Connection refused > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:98) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:56) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:57) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:175) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_65] > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-10 12:27:22,808 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-24) Command GetCapabilitiesVDSCommand(HostName = rack, HostId = 7ebdb254-efb6-480b-8c4e-a7dbdc759ed4, vds=Host[rack,7ebdb254-efb6-480b-8c4e-a7dbdc759ed4]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-09-10 12:27:22,808 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-24) Host rack is not responding. It will stay in Connecting state for a grace period of 60 seconds and after that an attempt to fence the host will be issued. >2014-09-10 12:27:22,943 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-24) Server failed to respond, vds_id = 7ebdb254-efb6-480b-8c4e-a7dbdc759ed4, vds_name = rack, vm_count = 0, spm_status = None, non-responsive_timeout (seconds) = 60, error = java.net.ConnectException: Connection refused >2014-09-10 12:27:22,962 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-24) Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) > at sun.nio.ch.Net.connect(Net.java:465) > at sun.nio.ch.Net.connect(Net.java:457) > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) >, Custom Event ID: -1, Message: Host rack is non responsive. >2014-09-10 12:27:23,230 INFO [org.ovirt.engine.core.bll.VdsEventListener] (org.ovirt.thread.pool-6-thread-4) ResourceManager::vdsNotResponding entered for Host 7ebdb254-efb6-480b-8c4e-a7dbdc759ed4, dell-r210ii-16.rhev.lab.eng.brq.redhat.com >2014-09-10 12:27:23,286 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-24) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_65] > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-10 12:27:23,516 INFO [org.ovirt.engine.core.bll.aaa.LoginAdminUserCommand] (ajp-/127.0.0.1:8702-4) Running command: LoginAdminUserCommand internal: false. >2014-09-10 12:27:23,704 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-4) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-09-10 12:27:23,764 INFO [org.ovirt.engine.core.bll.SshSoftFencingCommand] (org.ovirt.thread.pool-6-thread-4) [6236bbb6] Running command: SshSoftFencingCommand internal: true. Entities affected : ID: 7ebdb254-efb6-480b-8c4e-a7dbdc759ed4 Type: VDS >2014-09-10 12:27:23,772 INFO [org.ovirt.engine.core.bll.SshSoftFencingCommand] (org.ovirt.thread.pool-6-thread-4) [6236bbb6] Opening SSH Soft Fencing session on host dell-r210ii-16.rhev.lab.eng.brq.redhat.com >2014-09-10 12:27:24,283 ERROR [org.ovirt.engine.core.bll.SshSoftFencingCommand] (org.ovirt.thread.pool-6-thread-4) [6236bbb6] SSH Soft Fencing command failed on host dell-r210ii-16.rhev.lab.eng.brq.redhat.com: SSH session closed during connection 'root@dell-r210ii-16.rhev.lab.eng.brq.redhat.com' >Stdout: >Stderr: >Stacktrace: java.io.IOException: SSH session closed during connection 'root@dell-r210ii-16.rhev.lab.eng.brq.redhat.com': java.io.IOException: SSH session closed during connection 'root@dell-r210ii-16.rhev.lab.eng.brq.redhat.com' > at org.ovirt.engine.core.uutils.ssh.SSHClient.connect(SSHClient.java:309) [uutils.jar:] > at org.ovirt.engine.core.bll.utils.EngineSSHClient.connect(EngineSSHClient.java:59) [bll.jar:] > at org.ovirt.engine.core.bll.SshSoftFencingCommand.executeSshSoftFencingCommand(SshSoftFencingCommand.java:87) [bll.jar:] > at org.ovirt.engine.core.bll.SshSoftFencingCommand.executeCommand(SshSoftFencingCommand.java:53) [bll.jar:] > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1167) [bll.jar:] > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1298) [bll.jar:] > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1920) [bll.jar:] > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) [utils.jar:] > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) [utils.jar:] > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1319) [bll.jar:] > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:344) [bll.jar:] > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:430) [bll.jar:] > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:411) [bll.jar:] > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:616) [bll.jar:] > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_65] > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptor.java:52) [jboss-as-ee.jar:7.4.0.Final-redhat-19] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:63) [jboss-as-ee.jar:7.4.0.Final-redhat-19] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:374) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.as.weld.ejb.Jsr299BindingsInterceptor.delegateInterception(Jsr299BindingsInterceptor.java:74) [jboss-as-weld.jar:7.4.0.Final-redhat-19] > at org.jboss.as.weld.ejb.Jsr299BindingsInterceptor.doMethodInterception(Jsr299BindingsInterceptor.java:84) [jboss-as-weld.jar:7.4.0.Final-redhat-19] > at org.jboss.as.weld.ejb.Jsr299BindingsInterceptor.processInvocation(Jsr299BindingsInterceptor.java:97) [jboss-as-weld.jar:7.4.0.Final-redhat-19] > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:63) [jboss-as-ee.jar:7.4.0.Final-redhat-19] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:63) [jboss-as-ee.jar:7.4.0.Final-redhat-19] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) [jboss-as-ejb3.jar:7.4.0.Final-redhat-19] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.as.weld.ejb.EjbRequestScopeActivationInterceptor.processInvocation(EjbRequestScopeActivationInterceptor.java:93) [jboss-as-weld.jar:7.4.0.Final-redhat-19] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) [jboss-as-ee.jar:7.4.0.Final-redhat-19] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) [jboss-as-ejb3.jar:7.4.0.Final-redhat-19] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) [jboss-as-ejb3.jar:7.4.0.Final-redhat-19] > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) [jboss-as-ejb3.jar:7.4.0.Final-redhat-19] > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) [jboss-as-ejb3.jar:7.4.0.Final-redhat-19] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) [jboss-as-ejb3.jar:7.4.0.Final-redhat-19] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) [jboss-as-ejb3.jar:7.4.0.Final-redhat-19] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) [jboss-as-ejb3.jar:7.4.0.Final-redhat-19] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) [jboss-as-ee.jar:7.4.0.Final-redhat-19] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) [jboss-as-ee.jar:7.4.0.Final-redhat-19] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:185) [jboss-as-ee.jar:7.4.0.Final-redhat-19] > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) [jboss-as-ee.jar:7.4.0.Final-redhat-19] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:73) [jboss-as-ee.jar:7.4.0.Final-redhat-19] > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view7.runInternalAction(Unknown Source) [bll.jar:] > at org.ovirt.engine.core.bll.VdsEventListener$4.run(VdsEventListener.java:243) [bll.jar:] > at org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil$InternalWrapperRunnable.run(ThreadPoolUtil.java:90) [utils.jar:] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_65] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_65] > at java.lang.Thread.run(Thread.java:745) [rt.jar:1.7.0_65] > >2014-09-10 12:27:24,475 INFO [org.ovirt.engine.core.bll.VdsNotRespondingTreatmentCommand] (org.ovirt.thread.pool-6-thread-4) [4992ab4d] Lock Acquired to object EngineLock [exclusiveLocks= key: 7ebdb254-efb6-480b-8c4e-a7dbdc759ed4 value: VDS_FENCE >, sharedLocks= ] >2014-09-10 12:27:24,475 ERROR [org.ovirt.engine.core.bll.VdsNotRespondingTreatmentCommand] (org.ovirt.thread.pool-6-thread-4) [4992ab4d] Failed to run Fence script on vds:rack. >2014-09-10 12:27:24,514 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-4) [4992ab4d] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host rack became non responsive. It has no power management configured. Please check the host status, manually reboot it, and click "Confirm Host Has Been Rebooted" >2014-09-10 12:27:24,514 WARN [org.ovirt.engine.core.bll.VdsNotRespondingTreatmentCommand] (org.ovirt.thread.pool-6-thread-4) [4992ab4d] CanDoAction of action VdsNotRespondingTreatment failed. Reasons:VAR__ACTION__RESTART,VDS_FENCE_DISABLED >2014-09-10 12:27:24,514 INFO [org.ovirt.engine.core.bll.VdsNotRespondingTreatmentCommand] (org.ovirt.thread.pool-6-thread-4) [4992ab4d] Lock freed to object EngineLock [exclusiveLocks= key: 7ebdb254-efb6-480b-8c4e-a7dbdc759ed4 value: VDS_FENCE >, sharedLocks= ] >2014-09-10 12:27:26,352 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to dell-r210ii-16.rhev.lab.eng.brq.redhat.com/10.34.62.208 >2014-09-10 12:27:26,353 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-10 12:27:26,353 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-10 12:27:26,353 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-28) Exception during connection: java.util.concurrent.ExecutionException: java.net.ConnectException: Connection refused > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:98) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:56) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:57) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:175) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_65] > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-10 12:27:26,357 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-28) Command GetCapabilitiesVDSCommand(HostName = rack, HostId = 7ebdb254-efb6-480b-8c4e-a7dbdc759ed4, vds=Host[rack,7ebdb254-efb6-480b-8c4e-a7dbdc759ed4]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-09-10 12:27:26,358 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-28) Host rack is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-10 12:27:26,384 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-28) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_65] > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-10 12:27:27,303 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp-/127.0.0.1:8702-3) Running command: LoginUserCommand internal: false. >2014-09-10 12:27:29,410 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to dell-r210ii-16.rhev.lab.eng.brq.redhat.com/10.34.62.208 >2014-09-10 12:27:29,411 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-10 12:27:29,411 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-10 12:27:29,412 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-34) Exception during connection: java.util.concurrent.ExecutionException: java.net.ConnectException: Connection refused > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:98) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:56) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:57) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:175) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_65] > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-10 12:27:29,414 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-34) Command GetCapabilitiesVDSCommand(HostName = rack, HostId = 7ebdb254-efb6-480b-8c4e-a7dbdc759ed4, vds=Host[rack,7ebdb254-efb6-480b-8c4e-a7dbdc759ed4]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-09-10 12:27:29,414 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-34) Host rack is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-10 12:27:29,428 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-34) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_65] > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-10 12:27:32,496 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to dell-r210ii-16.rhev.lab.eng.brq.redhat.com/10.34.62.208 >2014-09-10 12:27:32,497 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-10 12:27:32,497 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-10 12:27:32,498 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-43) Exception during connection: java.util.concurrent.ExecutionException: java.net.ConnectException: Connection refused > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:98) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:56) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:57) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:175) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_65] > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-10 12:27:32,500 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-43) Command GetCapabilitiesVDSCommand(HostName = rack, HostId = 7ebdb254-efb6-480b-8c4e-a7dbdc759ed4, vds=Host[rack,7ebdb254-efb6-480b-8c4e-a7dbdc759ed4]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-09-10 12:27:32,500 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-43) Host rack is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-10 12:27:32,524 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-43) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_65] > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-10 12:27:35,550 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to dell-r210ii-16.rhev.lab.eng.brq.redhat.com/10.34.62.208 >2014-09-10 12:27:35,550 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-10 12:27:35,551 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-10 12:27:35,551 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-47) Exception during connection: java.util.concurrent.ExecutionException: java.net.ConnectException: Connection refused > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:98) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:56) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:57) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:175) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_65] > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-10 12:27:35,555 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-47) Command GetCapabilitiesVDSCommand(HostName = rack, HostId = 7ebdb254-efb6-480b-8c4e-a7dbdc759ed4, vds=Host[rack,7ebdb254-efb6-480b-8c4e-a7dbdc759ed4]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-09-10 12:27:35,557 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-47) Host rack is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-10 12:27:35,573 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-47) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_65] > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-10 12:27:38,589 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to dell-r210ii-16.rhev.lab.eng.brq.redhat.com/10.34.62.208 >2014-09-10 12:27:38,590 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-10 12:27:38,591 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-10 12:27:38,591 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-54) Exception during connection: java.util.concurrent.ExecutionException: java.net.ConnectException: Connection refused > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:98) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:56) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:57) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:175) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_65] > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-10 12:27:38,594 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-54) Command GetCapabilitiesVDSCommand(HostName = rack, HostId = 7ebdb254-efb6-480b-8c4e-a7dbdc759ed4, vds=Host[rack,7ebdb254-efb6-480b-8c4e-a7dbdc759ed4]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-09-10 12:27:38,594 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-54) Host rack is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-10 12:27:38,611 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-54) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_65] > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-10 12:27:41,629 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to dell-r210ii-16.rhev.lab.eng.brq.redhat.com/10.34.62.208 >2014-09-10 12:27:41,630 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-10 12:27:41,630 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-10 12:27:41,631 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-61) Exception during connection: java.util.concurrent.ExecutionException: java.net.ConnectException: Connection refused > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:98) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:56) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:57) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:175) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_65] > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-10 12:27:41,633 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-61) Command GetCapabilitiesVDSCommand(HostName = rack, HostId = 7ebdb254-efb6-480b-8c4e-a7dbdc759ed4, vds=Host[rack,7ebdb254-efb6-480b-8c4e-a7dbdc759ed4]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-09-10 12:27:41,633 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-61) Host rack is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-10 12:27:41,649 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-61) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_65] > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-10 12:27:44,671 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to dell-r210ii-16.rhev.lab.eng.brq.redhat.com/10.34.62.208 >2014-09-10 12:27:44,672 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-10 12:27:44,673 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-10 12:27:44,673 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-70) Exception during connection: java.util.concurrent.ExecutionException: java.net.ConnectException: Connection refused > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:98) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:56) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:57) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:175) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_65] > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-10 12:27:44,675 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-70) Command GetCapabilitiesVDSCommand(HostName = rack, HostId = 7ebdb254-efb6-480b-8c4e-a7dbdc759ed4, vds=Host[rack,7ebdb254-efb6-480b-8c4e-a7dbdc759ed4]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-09-10 12:27:44,676 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-70) Host rack is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-10 12:27:44,688 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-70) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_65] > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-10 12:27:46,648 INFO [org.ovirt.engine.core.bll.storage.FenceVdsManualyCommand] (ajp-/127.0.0.1:8702-2) [1e2731f0] Lock Acquired to object EngineLock [exclusiveLocks= key: 7ebdb254-efb6-480b-8c4e-a7dbdc759ed4 value: VDS_FENCE >, sharedLocks= ] >2014-09-10 12:27:46,660 INFO [org.ovirt.engine.core.bll.storage.FenceVdsManualyCommand] (org.ovirt.thread.pool-6-thread-15) [1e2731f0] Running command: FenceVdsManualyCommand internal: false. Entities affected : ID: 7ebdb254-efb6-480b-8c4e-a7dbdc759ed4 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2014-09-10 12:27:46,686 INFO [org.ovirt.engine.core.bll.ClearNonResponsiveVdsVmsCommand] (org.ovirt.thread.pool-6-thread-15) [22cd7068] Running command: ClearNonResponsiveVdsVmsCommand internal: true. Entities affected : ID: 7ebdb254-efb6-480b-8c4e-a7dbdc759ed4 Type: VDS >2014-09-10 12:27:46,697 INFO [org.ovirt.engine.core.vdsbroker.UpdateVdsVMsClearedVDSCommand] (org.ovirt.thread.pool-6-thread-15) [22cd7068] START, UpdateVdsVMsClearedVDSCommand(HostName = rack, HostId = 7ebdb254-efb6-480b-8c4e-a7dbdc759ed4), log id: 44748c >2014-09-10 12:27:46,716 INFO [org.ovirt.engine.core.vdsbroker.UpdateVdsVMsClearedVDSCommand] (org.ovirt.thread.pool-6-thread-15) [22cd7068] FINISH, UpdateVdsVMsClearedVDSCommand, log id: 44748c >2014-09-10 12:27:46,746 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-15) [22cd7068] Correlation ID: 22cd7068, Call Stack: null, Custom Event ID: -1, Message: All VMs' status on Non Responsive Host rack were changed to 'Down' by admin >2014-09-10 12:27:46,780 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-15) [22cd7068] Correlation ID: 1e2731f0, Call Stack: null, Custom Event ID: -1, Message: Manual fence for host rack was started. >2014-09-10 12:27:46,781 INFO [org.ovirt.engine.core.bll.storage.FenceVdsManualyCommand] (org.ovirt.thread.pool-6-thread-15) [22cd7068] Lock freed to object EngineLock [exclusiveLocks= key: 7ebdb254-efb6-480b-8c4e-a7dbdc759ed4 value: VDS_FENCE >, sharedLocks= ] >2014-09-10 12:27:47,713 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to dell-r210ii-16.rhev.lab.eng.brq.redhat.com/10.34.62.208 >2014-09-10 12:27:47,717 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-10 12:27:47,717 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-10 12:27:47,717 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-76) Exception during connection: java.util.concurrent.ExecutionException: java.net.ConnectException: Connection refused > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:98) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:56) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:57) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:175) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_65] > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-10 12:27:47,720 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-76) Command GetCapabilitiesVDSCommand(HostName = rack, HostId = 7ebdb254-efb6-480b-8c4e-a7dbdc759ed4, vds=Host[rack,7ebdb254-efb6-480b-8c4e-a7dbdc759ed4]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-09-10 12:27:47,720 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-76) Host rack is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-10 12:27:47,748 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-76) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_65] > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-10 12:27:50,777 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to dell-r210ii-16.rhev.lab.eng.brq.redhat.com/10.34.62.208 >2014-09-10 12:27:50,778 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-10 12:27:50,779 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-10 12:27:50,779 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-80) Exception during connection: java.util.concurrent.ExecutionException: java.net.ConnectException: Connection refused > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:98) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:56) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:57) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:175) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_65] > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-10 12:27:50,784 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-80) Command GetCapabilitiesVDSCommand(HostName = rack, HostId = 7ebdb254-efb6-480b-8c4e-a7dbdc759ed4, vds=Host[rack,7ebdb254-efb6-480b-8c4e-a7dbdc759ed4]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-09-10 12:27:50,784 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-80) Host rack is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-10 12:27:50,810 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-80) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_65] > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-10 12:27:53,834 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to dell-r210ii-16.rhev.lab.eng.brq.redhat.com/10.34.62.208 >2014-09-10 12:27:53,835 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-10 12:27:53,835 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-10 12:27:53,836 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-90) Exception during connection: java.util.concurrent.ExecutionException: java.net.ConnectException: Connection refused > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:98) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:56) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:57) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:175) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_65] > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-10 12:27:53,838 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-90) Command GetCapabilitiesVDSCommand(HostName = rack, HostId = 7ebdb254-efb6-480b-8c4e-a7dbdc759ed4, vds=Host[rack,7ebdb254-efb6-480b-8c4e-a7dbdc759ed4]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-09-10 12:27:53,838 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-90) Host rack is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-10 12:27:53,860 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-90) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_65] > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-10 12:27:56,881 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to dell-r210ii-16.rhev.lab.eng.brq.redhat.com/10.34.62.208 >2014-09-10 12:27:56,882 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-10 12:27:56,882 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-10 12:27:56,883 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-94) Exception during connection: java.util.concurrent.ExecutionException: java.net.ConnectException: Connection refused > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:98) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:56) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:57) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:175) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_65] > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-10 12:27:56,888 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-94) Command GetCapabilitiesVDSCommand(HostName = rack, HostId = 7ebdb254-efb6-480b-8c4e-a7dbdc759ed4, vds=Host[rack,7ebdb254-efb6-480b-8c4e-a7dbdc759ed4]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-09-10 12:27:56,888 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-94) Host rack is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-10 12:27:56,917 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-94) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_65] > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-10 12:27:59,943 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to dell-r210ii-16.rhev.lab.eng.brq.redhat.com/10.34.62.208 >2014-09-10 12:27:59,944 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-10 12:27:59,945 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-10 12:27:59,945 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-100) Exception during connection: java.util.concurrent.ExecutionException: java.net.ConnectException: Connection refused > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:98) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:56) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:57) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:175) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_65] > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-10 12:27:59,947 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-100) Command GetCapabilitiesVDSCommand(HostName = rack, HostId = 7ebdb254-efb6-480b-8c4e-a7dbdc759ed4, vds=Host[rack,7ebdb254-efb6-480b-8c4e-a7dbdc759ed4]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-09-10 12:27:59,948 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-100) Host rack is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-10 12:27:59,960 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-100) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_65] > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-10 12:28:02,973 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to dell-r210ii-16.rhev.lab.eng.brq.redhat.com/10.34.62.208 >2014-09-10 12:28:02,974 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-10 12:28:02,974 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-10 12:28:02,975 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-10) Exception during connection: java.util.concurrent.ExecutionException: java.net.ConnectException: Connection refused > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:98) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:56) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:57) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:175) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_65] > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-10 12:28:02,977 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-10) Command GetCapabilitiesVDSCommand(HostName = rack, HostId = 7ebdb254-efb6-480b-8c4e-a7dbdc759ed4, vds=Host[rack,7ebdb254-efb6-480b-8c4e-a7dbdc759ed4]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-09-10 12:28:02,977 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-10) Host rack is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-10 12:28:02,992 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-10) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_65] > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-10 12:28:06,023 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to dell-r210ii-16.rhev.lab.eng.brq.redhat.com/10.34.62.208 >2014-09-10 12:28:06,024 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-10 12:28:06,024 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-10 12:28:06,024 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-14) Exception during connection: java.util.concurrent.ExecutionException: java.net.ConnectException: Connection refused > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:98) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:56) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:57) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:175) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor45.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-10 12:28:06,041 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-14) Command GetCapabilitiesVDSCommand(HostName = rack, HostId = 7ebdb254-efb6-480b-8c4e-a7dbdc759ed4, vds=Host[rack,7ebdb254-efb6-480b-8c4e-a7dbdc759ed4]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-09-10 12:28:06,041 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-14) Host rack is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-10 12:28:06,053 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-14) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor45.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-10 12:28:09,067 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to dell-r210ii-16.rhev.lab.eng.brq.redhat.com/10.34.62.208 >2014-09-10 12:28:09,067 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-10 12:28:09,068 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-10 12:28:09,068 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-21) Exception during connection: java.util.concurrent.ExecutionException: java.net.ConnectException: Connection refused > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:98) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:56) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:57) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:175) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor45.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-10 12:28:09,071 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-21) Command GetCapabilitiesVDSCommand(HostName = rack, HostId = 7ebdb254-efb6-480b-8c4e-a7dbdc759ed4, vds=Host[rack,7ebdb254-efb6-480b-8c4e-a7dbdc759ed4]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-09-10 12:28:09,071 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-21) Host rack is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-10 12:28:09,091 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-21) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor45.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-10 12:28:11,312 INFO [org.ovirt.engine.core.bll.MaintenanceNumberOfVdssCommand] (org.ovirt.thread.pool-6-thread-24) [cd923b] Running command: MaintenanceNumberOfVdssCommand internal: false. Entities affected : ID: 7ebdb254-efb6-480b-8c4e-a7dbdc759ed4 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2014-09-10 12:28:11,467 INFO [org.ovirt.engine.core.bll.MaintenanceVdsCommand] (org.ovirt.thread.pool-6-thread-24) [cd923b] Running command: MaintenanceVdsCommand internal: true. Entities affected : ID: 7ebdb254-efb6-480b-8c4e-a7dbdc759ed4 Type: VDS >2014-09-10 12:28:11,474 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-6-thread-24) [cd923b] START, SetVdsStatusVDSCommand(HostName = rack, HostId = 7ebdb254-efb6-480b-8c4e-a7dbdc759ed4, status=Maintenance, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 25adf1bf >2014-09-10 12:28:11,503 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-6-thread-24) [cd923b] FINISH, SetVdsStatusVDSCommand, log id: 25adf1bf >2014-09-10 12:28:11,529 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-24) [cd923b] Correlation ID: cd923b, Job ID: 0e29fc1f-4548-487d-94bf-1061f25e104e, Call Stack: null, Custom Event ID: -1, Message: Host rack was switched to Maintenance mode by admin. >2014-09-10 12:28:12,103 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-34) Initializing Host: rack >2014-09-10 12:28:22,518 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp-/127.0.0.1:8702-9) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-09-10 12:28:22,519 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp-/127.0.0.1:8702-9) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-09-10 12:28:22,519 INFO [org.ovirt.engine.core.bll.InstallVdsCommand] (ajp-/127.0.0.1:8702-9) [46799a9e] Running command: InstallVdsCommand internal: false. Entities affected : ID: 7ebdb254-efb6-480b-8c4e-a7dbdc759ed4 Type: VDSAction group EDIT_HOST_CONFIGURATION with role type ADMIN >2014-09-10 12:28:22,669 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp-/127.0.0.1:8702-9) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-09-10 12:28:22,669 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp-/127.0.0.1:8702-9) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-09-10 12:28:22,677 INFO [org.ovirt.engine.core.bll.InstallVdsInternalCommand] (ajp-/127.0.0.1:8702-9) [46799a9e] Lock Acquired to object EngineLock [exclusiveLocks= key: 7ebdb254-efb6-480b-8c4e-a7dbdc759ed4 value: VDS >, sharedLocks= ] >2014-09-10 12:28:22,692 INFO [org.ovirt.engine.core.bll.InstallVdsInternalCommand] (org.ovirt.thread.pool-6-thread-25) [46799a9e] Running command: InstallVdsInternalCommand internal: true. Entities affected : ID: 7ebdb254-efb6-480b-8c4e-a7dbdc759ed4 Type: VDS >2014-09-10 12:28:22,699 INFO [org.ovirt.engine.core.bll.InstallVdsInternalCommand] (org.ovirt.thread.pool-6-thread-25) [46799a9e] Before Installation host 7ebdb254-efb6-480b-8c4e-a7dbdc759ed4, rack >2014-09-10 12:28:22,702 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-9) [46799a9e] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host rack. >2014-09-10 12:28:22,746 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-6-thread-25) [46799a9e] START, SetVdsStatusVDSCommand(HostName = rack, HostId = 7ebdb254-efb6-480b-8c4e-a7dbdc759ed4, status=Installing, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 3abd3632 >2014-09-10 12:28:22,747 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-9) [46799a9e] Correlation ID: 46799a9e, Call Stack: null, Custom Event ID: -1, Message: Host rack configuration was updated by admin. >2014-09-10 12:28:22,779 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-6-thread-25) [46799a9e] FINISH, SetVdsStatusVDSCommand, log id: 3abd3632 >2014-09-10 12:28:22,875 INFO [org.ovirt.engine.core.bll.InstallerMessages] (org.ovirt.thread.pool-6-thread-25) [46799a9e] Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Connected to host dell-r210ii-16.rhev.lab.eng.brq.redhat.com with SSH key fingerprint: 25:12:7b:de:30:1d:b3:5e:bf:d2:a6:c7:37:77:d3:b1 >2014-09-10 12:28:22,940 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-25) [46799a9e] Correlation ID: 46799a9e, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Connected to host dell-r210ii-16.rhev.lab.eng.brq.redhat.com with SSH key fingerprint: 25:12:7b:de:30:1d:b3:5e:bf:d2:a6:c7:37:77:d3:b1. >2014-09-10 12:28:22,984 INFO [org.ovirt.engine.core.bll.VdsDeploy] (org.ovirt.thread.pool-6-thread-25) [46799a9e] Installation of dell-r210ii-16.rhev.lab.eng.brq.redhat.com. Executing command via SSH umask 0077; MYTMP="$(TMPDIR="${OVIRT_TMPDIR}" mktemp -t ovirt-XXXXXXXXXX)"; trap "chmod -R u+rwX \"${MYTMP}\" > /dev/null 2>&1; rm -fr \"${MYTMP}\" > /dev/null 2>&1" 0; rm -fr "${MYTMP}" && mkdir "${MYTMP}" && tar --warning=no-timestamp -C "${MYTMP}" -x && "${MYTMP}"/setup DIALOG/dialect=str:machine DIALOG/customization=bool:True < /var/cache/ovirt-engine/ovirt-host-deploy.tar >2014-09-10 12:28:23,006 INFO [org.ovirt.engine.core.uutils.ssh.SSHDialog] (org.ovirt.thread.pool-6-thread-25) SSH execute root@dell-r210ii-16.rhev.lab.eng.brq.redhat.com 'umask 0077; MYTMP="$(TMPDIR="${OVIRT_TMPDIR}" mktemp -t ovirt-XXXXXXXXXX)"; trap "chmod -R u+rwX \"${MYTMP}\" > /dev/null 2>&1; rm -fr \"${MYTMP}\" > /dev/null 2>&1" 0; rm -fr "${MYTMP}" && mkdir "${MYTMP}" && tar --warning=no-timestamp -C "${MYTMP}" -x && "${MYTMP}"/setup DIALOG/dialect=str:machine DIALOG/customization=bool:True' >2014-09-10 12:28:24,066 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Stage: Initializing >2014-09-10 12:28:24,093 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 46799a9e, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Stage: Initializing. >2014-09-10 12:28:24,108 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Stage: Environment setup >2014-09-10 12:28:24,143 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 46799a9e, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Stage: Environment setup. >2014-09-10 12:28:24,144 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Stage: Environment packages setup >2014-09-10 12:28:24,158 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 46799a9e, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Stage: Environment packages setup. >2014-09-10 12:28:31,295 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Stage: Programs detection >2014-09-10 12:28:31,311 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 46799a9e, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Stage: Programs detection. >2014-09-10 12:28:31,311 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Stage: Environment customization >2014-09-10 12:28:31,329 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 46799a9e, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Stage: Environment customization. >2014-09-10 12:28:31,727 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Kdump supported >2014-09-10 12:28:31,743 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 46799a9e, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Kdump supported. >2014-09-10 12:28:31,762 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Logs at host located at: '/tmp/ovirt-host-deploy-20140910122823-iw36ig.log' >2014-09-10 12:28:31,777 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 46799a9e, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Logs at host located at: '/tmp/ovirt-host-deploy-20140910122823-iw36ig.log'. >2014-09-10 12:28:31,883 INFO [org.ovirt.engine.core.bll.VdsDeploy] (VdsDeploy) Host dell-r210ii-16.rhev.lab.eng.brq.redhat.com reports unique id 4C4C4544-0034-5310-8052-B4C04F4A354A >2014-09-10 12:28:31,953 INFO [org.ovirt.engine.core.bll.VdsDeploy] (VdsDeploy) Assigning unique id 4C4C4544-0034-5310-8052-B4C04F4A354A to Host dell-r210ii-16.rhev.lab.eng.brq.redhat.com >2014-09-10 12:28:32,077 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Disabling Kdump integration >2014-09-10 12:28:32,090 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 46799a9e, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Disabling Kdump integration. >2014-09-10 12:28:32,127 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Stage: Setup validation >2014-09-10 12:28:32,145 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 46799a9e, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Stage: Setup validation. >2014-09-10 12:28:32,175 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Hardware supports virtualization >2014-09-10 12:28:32,186 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 46799a9e, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Hardware supports virtualization. >2014-09-10 12:28:32,413 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Stage: Transaction setup >2014-09-10 12:28:32,433 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 46799a9e, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Stage: Transaction setup. >2014-09-10 12:28:32,461 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Stage: Misc configuration >2014-09-10 12:28:32,471 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 46799a9e, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Stage: Misc configuration. >2014-09-10 12:28:32,471 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Stage: Package installation >2014-09-10 12:28:32,479 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 46799a9e, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Stage: Package installation. >2014-09-10 12:28:34,505 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Stage: Misc configuration >2014-09-10 12:28:34,522 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 46799a9e, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Stage: Misc configuration. >2014-09-10 12:28:34,523 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Setting time >2014-09-10 12:28:34,533 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 46799a9e, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Setting time. >2014-09-10 12:28:36,002 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Enrolling certificate >2014-09-10 12:28:36,017 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 46799a9e, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Enrolling certificate. >2014-09-10 12:28:37,288 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Stage: Transaction commit >2014-09-10 12:28:37,303 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 46799a9e, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Stage: Transaction commit. >2014-09-10 12:28:37,355 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Stage: Closing up >2014-09-10 12:28:37,367 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 46799a9e, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Stage: Closing up. >2014-09-10 12:28:38,134 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Stopping libvirtd >2014-09-10 12:28:38,146 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 46799a9e, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Stopping libvirtd. >2014-09-10 12:28:38,424 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Starting vdsm >2014-09-10 12:28:38,459 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 46799a9e, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Starting vdsm. >2014-09-10 12:28:40,225 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Stage: Pre-termination >2014-09-10 12:28:40,243 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 46799a9e, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Stage: Pre-termination. >2014-09-10 12:28:40,265 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Retrieving installation logs to: '/var/log/ovirt-engine/host-deploy/ovirt-20140910122840-dell-r210ii-16.rhev.lab.eng.brq.redhat.com-46799a9e.log' >2014-09-10 12:28:40,280 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 46799a9e, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Retrieving installation logs to: '/var/log/ovirt-engine/host-deploy/ovirt-20140910122840-dell-r210ii-16.rhev.lab.eng.brq.redhat.com-46799a9e.log'. >2014-09-10 12:28:40,527 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Stage: Termination >2014-09-10 12:28:40,545 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 46799a9e, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Stage: Termination. >2014-09-10 12:28:40,665 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to dell-r210ii-16.rhev.lab.eng.brq.redhat.com/10.34.62.208 >2014-09-10 12:28:41,038 INFO [org.ovirt.engine.core.bll.network.NetworkConfigurator] (org.ovirt.thread.pool-6-thread-25) [46799a9e] Engine managed to communicate with VDSM agent on host rack >2014-09-10 12:28:41,575 INFO [org.ovirt.engine.core.bll.network.host.SetupNetworksCommand] (org.ovirt.thread.pool-6-thread-25) [1be7dae4] Running command: SetupNetworksCommand internal: true. Entities affected : ID: 7ebdb254-efb6-480b-8c4e-a7dbdc759ed4 Type: VDSAction group CONFIGURE_HOST_NETWORK with role type ADMIN >2014-09-10 12:28:41,605 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetupNetworksVDSCommand] (org.ovirt.thread.pool-6-thread-25) [1be7dae4] START, SetupNetworksVDSCommand(HostName = rack, HostId = 7ebdb254-efb6-480b-8c4e-a7dbdc759ed4, force=false, checkConnectivity=true, conectivityTimeout=120, > networks=[rhevm {id=00000000-0000-0000-0000-000000000009, description=Management Network, comment=null, subnet=null, gateway=null, type=null, vlanId=null, stp=false, dataCenterId=00000002-0002-0002-0002-0000000001ad, mtu=0, vmNetwork=true, cluster=NetworkCluster {id={clusterId=null, networkId=null}, status=OPERATIONAL, display=true, required=true, migration=true}, providedBy=null, label=null, qosId=null}], > bonds=[], > interfaces=[em1 {id=c97613e8-b860-495b-9207-7715533de9eb, vdsId=7ebdb254-efb6-480b-8c4e-a7dbdc759ed4, macAddress=d4:ae:52:c7:0d:8b, networkName=rhevm, baseInterface=null, vlanId=null, bonded=null, bondName=null, bondOptions=null, bootProtocol=DHCP, address=10.34.62.208, subnet=255.255.252.0, gateway=null, mtu=1500, bridged=false, speed=1000, type=0, networkImplementationDetails=null, qos=null qosOverridden=false, labels=null, customProperties=null}, > em2 {id=ca117a0b-8f32-4c38-8b4c-aa15bd19f786, vdsId=7ebdb254-efb6-480b-8c4e-a7dbdc759ed4, macAddress=d4:ae:52:c7:0d:8c, networkName=null, baseInterface=null, vlanId=null, bonded=null, bondName=null, bondOptions=null, bootProtocol=DHCP, address=, subnet=, gateway=null, mtu=1500, bridged=false, speed=0, type=0, networkImplementationDetails=null, qos=null qosOverridden=false, labels=null, customProperties=null}], > removedNetworks=[], > removedBonds=[]), log id: 5637f29e >2014-09-10 12:28:41,619 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetupNetworksVDSCommand] (org.ovirt.thread.pool-6-thread-25) [1be7dae4] FINISH, SetupNetworksVDSCommand, log id: 5637f29e >2014-09-10 12:30:44,135 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp-/127.0.0.1:8702-11) Running command: LoginUserCommand internal: false. >2014-09-10 12:30:44,145 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-11) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-09-10 12:30:44,808 INFO [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp-/127.0.0.1:8702-11) [6fb0c3a2] Running command: LogoutBySessionCommand internal: false. >2014-09-10 12:30:44,831 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp-/127.0.0.1:8702-11) [515f938] Running command: LogoutUserCommand internal: false. >2014-09-10 12:30:44,844 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-11) [515f938] Correlation ID: 515f938, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-09-10 12:31:06,839 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp-/127.0.0.1:8702-9) Running command: LoginUserCommand internal: false. >2014-09-10 12:31:06,849 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-9) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-09-10 12:31:06,876 INFO [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp-/127.0.0.1:8702-9) [7cfa7bf9] Running command: LogoutBySessionCommand internal: false. >2014-09-10 12:31:06,881 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp-/127.0.0.1:8702-9) [2bdae329] Running command: LogoutUserCommand internal: false. >2014-09-10 12:31:06,892 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-9) [2bdae329] Correlation ID: 2bdae329, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-09-10 12:31:20,849 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp-/127.0.0.1:8702-4) Running command: LoginUserCommand internal: false. >2014-09-10 12:31:20,860 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-4) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-09-10 12:31:22,473 INFO [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp-/127.0.0.1:8702-4) [4d34c5fc] Running command: LogoutBySessionCommand internal: false. >2014-09-10 12:31:22,481 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp-/127.0.0.1:8702-4) [33e6a820] Running command: LogoutUserCommand internal: false. >2014-09-10 12:31:22,489 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-4) [33e6a820] Correlation ID: 33e6a820, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-09-10 12:31:41,802 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SetupNetworksVDSCommand] (org.ovirt.thread.pool-6-thread-25) [1be7dae4] org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues' >2014-09-10 12:31:41,804 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SetupNetworksVDSCommand] (org.ovirt.thread.pool-6-thread-25) [1be7dae4] Command SetupNetworksVDSCommand(HostName = rack, HostId = 7ebdb254-efb6-480b-8c4e-a7dbdc759ed4, force=false, checkConnectivity=true, conectivityTimeout=120, > networks=[rhevm {id=00000000-0000-0000-0000-000000000009, description=Management Network, comment=null, subnet=null, gateway=null, type=null, vlanId=null, stp=false, dataCenterId=00000002-0002-0002-0002-0000000001ad, mtu=0, vmNetwork=true, cluster=NetworkCluster {id={clusterId=null, networkId=null}, status=OPERATIONAL, display=true, required=true, migration=true}, providedBy=null, label=null, qosId=null}], > bonds=[], > interfaces=[em1 {id=c97613e8-b860-495b-9207-7715533de9eb, vdsId=7ebdb254-efb6-480b-8c4e-a7dbdc759ed4, macAddress=d4:ae:52:c7:0d:8b, networkName=rhevm, baseInterface=null, vlanId=null, bonded=null, bondName=null, bondOptions=null, bootProtocol=DHCP, address=10.34.62.208, subnet=255.255.252.0, gateway=null, mtu=1500, bridged=false, speed=1000, type=0, networkImplementationDetails=null, qos=null qosOverridden=false, labels=null, customProperties=null}, > em2 {id=ca117a0b-8f32-4c38-8b4c-aa15bd19f786, vdsId=7ebdb254-efb6-480b-8c4e-a7dbdc759ed4, macAddress=d4:ae:52:c7:0d:8c, networkName=null, baseInterface=null, vlanId=null, bonded=null, bondName=null, bondOptions=null, bootProtocol=DHCP, address=, subnet=, gateway=null, mtu=1500, bridged=false, speed=0, type=0, networkImplementationDetails=null, qos=null qosOverridden=false, labels=null, customProperties=null}], > removedNetworks=[], > removedBonds=[]) execution failed. Exception: VDSNetworkException: VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues' >2014-09-10 12:31:41,804 ERROR [org.ovirt.engine.core.bll.network.host.SetupNetworksCommand] (org.ovirt.thread.pool-6-thread-25) [1be7dae4] Command org.ovirt.engine.core.bll.network.host.SetupNetworksCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues' (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-09-10 12:31:41,823 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-25) [1be7dae4] Failed to configure management network: Failed to configure management network on host rack due to setup networks failure. >2014-09-10 12:31:41,828 ERROR [org.ovirt.engine.core.bll.InstallVdsInternalCommand] (org.ovirt.thread.pool-6-thread-25) [1be7dae4] Host installation failed for host 7ebdb254-efb6-480b-8c4e-a7dbdc759ed4, rack.: org.ovirt.engine.core.bll.VdsCommand$VdsInstallException: Failed to configure management network on the host > at org.ovirt.engine.core.bll.InstallVdsInternalCommand.configureManagementNetwork(InstallVdsInternalCommand.java:206) [bll.jar:] > at org.ovirt.engine.core.bll.InstallVdsInternalCommand.installHost(InstallVdsInternalCommand.java:164) [bll.jar:] > at org.ovirt.engine.core.bll.InstallVdsInternalCommand.executeCommand(InstallVdsInternalCommand.java:78) [bll.jar:] > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1167) [bll.jar:] > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1298) [bll.jar:] > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1920) [bll.jar:] > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) [utils.jar:] > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) [utils.jar:] > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1319) [bll.jar:] > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:344) [bll.jar:] > at org.ovirt.engine.core.bll.MultipleActionsRunner.executeValidatedCommand(MultipleActionsRunner.java:191) [bll.jar:] > at org.ovirt.engine.core.bll.MultipleActionsRunner.runCommands(MultipleActionsRunner.java:158) [bll.jar:] > at org.ovirt.engine.core.bll.MultipleActionsRunner$2.run(MultipleActionsRunner.java:167) [bll.jar:] > at org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil$InternalWrapperRunnable.run(ThreadPoolUtil.java:90) [utils.jar:] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_65] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_65] > at java.lang.Thread.run(Thread.java:745) [rt.jar:1.7.0_65] >Caused by: org.ovirt.engine.core.bll.network.NetworkConfigurator$NetworkConfiguratorException: Failed to configure management network > at org.ovirt.engine.core.bll.network.NetworkConfigurator.configureManagementNetwork(NetworkConfigurator.java:255) [bll.jar:] > at org.ovirt.engine.core.bll.network.NetworkConfigurator.createManagementNetworkIfRequired(NetworkConfigurator.java:82) [bll.jar:] > at org.ovirt.engine.core.bll.InstallVdsInternalCommand.configureManagementNetwork(InstallVdsInternalCommand.java:198) [bll.jar:] > ... 18 more > >2014-09-10 12:31:41,832 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-6-thread-25) [1be7dae4] START, SetVdsStatusVDSCommand(HostName = rack, HostId = 7ebdb254-efb6-480b-8c4e-a7dbdc759ed4, status=NonOperational, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 1c72fb92 >2014-09-10 12:31:41,852 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-6-thread-25) [1be7dae4] FINISH, SetVdsStatusVDSCommand, log id: 1c72fb92 >2014-09-10 12:31:41,872 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-25) [1be7dae4] Correlation ID: 46799a9e, Call Stack: null, Custom Event ID: -1, Message: Host rack installation failed. Failed to configure management network on the host. >2014-09-10 12:31:41,872 INFO [org.ovirt.engine.core.bll.InstallVdsInternalCommand] (org.ovirt.thread.pool-6-thread-25) [1be7dae4] Lock freed to object EngineLock [exclusiveLocks= key: 7ebdb254-efb6-480b-8c4e-a7dbdc759ed4 value: VDS >, sharedLocks= ] >2014-09-10 12:35:00,041 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-50) Autorecovering 1 hosts >2014-09-10 12:35:00,041 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-50) Autorecovering hosts id: 7ebdb254-efb6-480b-8c4e-a7dbdc759ed4, name : rack >2014-09-10 12:35:00,050 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-50) [5b5b66dc] Lock Acquired to object EngineLock [exclusiveLocks= key: 7ebdb254-efb6-480b-8c4e-a7dbdc759ed4 value: VDS >, sharedLocks= ] >2014-09-10 12:35:00,050 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-50) [5b5b66dc] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 7ebdb254-efb6-480b-8c4e-a7dbdc759ed4 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2014-09-10 12:35:00,051 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-50) [5b5b66dc] Before acquiring lock in order to prevent monitoring for host rack from data-center Default >2014-09-10 12:35:00,051 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-50) [5b5b66dc] Lock acquired, from now a monitoring of host will be skipped for host rack from data-center Default >2014-09-10 12:35:00,205 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-50) [5b5b66dc] START, SetVdsStatusVDSCommand(HostName = rack, HostId = 7ebdb254-efb6-480b-8c4e-a7dbdc759ed4, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 6ad76cbd >2014-09-10 12:35:00,232 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-50) [5b5b66dc] FINISH, SetVdsStatusVDSCommand, log id: 6ad76cbd >2014-09-10 12:35:00,251 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-50) [5b5b66dc] Activate finished. Lock released. Monitoring can run now for host rack from data-center Default >2014-09-10 12:35:00,288 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-50) [5b5b66dc] Correlation ID: 5b5b66dc, Call Stack: null, Custom Event ID: -1, Message: Host rack was autorecovered. >2014-09-10 12:35:00,289 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-50) [5b5b66dc] Lock freed to object EngineLock [exclusiveLocks= key: 7ebdb254-efb6-480b-8c4e-a7dbdc759ed4 value: VDS >, sharedLocks= ] >2014-09-10 12:35:00,731 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to dell-r210ii-16.rhev.lab.eng.brq.redhat.com/10.34.62.208 >2014-09-10 12:35:00,735 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.Reactor] (SSL Stomp Reactor) Unable to process messages: java.nio.channels.ClosedChannelException > at sun.nio.ch.SocketChannelImpl.ensureReadOpen(SocketChannelImpl.java:252) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:295) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.SSLEngineNioHelper.read(SSLEngineNioHelper.java:51) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.SSLClient.read(SSLClient.java:81) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.readBuffer(ReactorClient.java:203) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.stomp.StompCommonClient.processIncoming(StompCommonClient.java:90) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.process(ReactorClient.java:144) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.SSLClient.process(SSLClient.java:115) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.processChannels(Reactor.java:86) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:62) [vdsm-jsonrpc-java-client.jar:] > >2014-09-10 12:38:02,936 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-52) Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@5944c12d >2014-09-10 12:38:02,937 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-52) HostName = rack >2014-09-10 12:38:02,937 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-52) Command GetCapabilitiesVDSCommand(HostName = rack, HostId = 7ebdb254-efb6-480b-8c4e-a7dbdc759ed4, vds=Host[rack,7ebdb254-efb6-480b-8c4e-a7dbdc759ed4]) execution failed. Exception: VDSNetworkException: VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues' >2014-09-10 12:38:02,937 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-52) Host rack is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-10 12:38:02,951 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-52) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host rack is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-10 12:38:03,007 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-52) Server failed to respond, vds_id = 7ebdb254-efb6-480b-8c4e-a7dbdc759ed4, vds_name = rack, vm_count = 0, spm_status = None, non-responsive_timeout (seconds) = 120, error = VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues' >2014-09-10 12:38:03,019 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-52) Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues' > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:183) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:16) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor45.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Host rack is non responsive. >2014-09-10 12:38:03,118 INFO [org.ovirt.engine.core.bll.VdsEventListener] (org.ovirt.thread.pool-6-thread-32) ResourceManager::vdsNotResponding entered for Host 7ebdb254-efb6-480b-8c4e-a7dbdc759ed4, dell-r210ii-16.rhev.lab.eng.brq.redhat.com >2014-09-10 12:38:03,135 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-52) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues' > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:183) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:16) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor45.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] > >2014-09-10 12:38:03,162 INFO [org.ovirt.engine.core.bll.VdsNotRespondingTreatmentCommand] (org.ovirt.thread.pool-6-thread-32) [62d854ff] Lock Acquired to object EngineLock [exclusiveLocks= key: 7ebdb254-efb6-480b-8c4e-a7dbdc759ed4 value: VDS_FENCE >, sharedLocks= ] >2014-09-10 12:38:03,162 ERROR [org.ovirt.engine.core.bll.VdsNotRespondingTreatmentCommand] (org.ovirt.thread.pool-6-thread-32) [62d854ff] Failed to run Fence script on vds:rack. >2014-09-10 12:38:03,182 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-32) [62d854ff] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host rack became non responsive. It has no power management configured. Please check the host status, manually reboot it, and click "Confirm Host Has Been Rebooted" >2014-09-10 12:38:03,182 WARN [org.ovirt.engine.core.bll.VdsNotRespondingTreatmentCommand] (org.ovirt.thread.pool-6-thread-32) [62d854ff] CanDoAction of action VdsNotRespondingTreatment failed. Reasons:VAR__ACTION__RESTART,VDS_FENCE_DISABLED >2014-09-10 12:38:03,182 INFO [org.ovirt.engine.core.bll.VdsNotRespondingTreatmentCommand] (org.ovirt.thread.pool-6-thread-32) [62d854ff] Lock freed to object EngineLock [exclusiveLocks= key: 7ebdb254-efb6-480b-8c4e-a7dbdc759ed4 value: VDS_FENCE >, sharedLocks= ] >2014-09-10 12:38:06,163 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to dell-r210ii-16.rhev.lab.eng.brq.redhat.com/10.34.62.208 >2014-09-10 12:38:06,164 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.Reactor] (SSL Stomp Reactor) Unable to process messages: java.nio.channels.ClosedChannelException > at sun.nio.ch.SocketChannelImpl.ensureWriteOpen(SocketChannelImpl.java:265) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:474) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.SSLEngineNioHelper.write(SSLEngineNioHelper.java:104) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.SSLEngineNioHelper.process(SSLEngineNioHelper.java:125) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.SSLClient.pendingOperations(SSLClient.java:74) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.SSLClient.process(SSLClient.java:98) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.processChannels(Reactor.java:86) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:62) [vdsm-jsonrpc-java-client.jar:] > >2014-09-10 12:41:08,502 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-16) Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@14e1d12a >2014-09-10 12:41:08,503 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-16) HostName = rack >2014-09-10 12:41:08,503 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-16) Command GetCapabilitiesVDSCommand(HostName = rack, HostId = 7ebdb254-efb6-480b-8c4e-a7dbdc759ed4, vds=Host[rack,7ebdb254-efb6-480b-8c4e-a7dbdc759ed4]) execution failed. Exception: VDSNetworkException: VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues' >2014-09-10 12:41:08,503 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-16) Host rack is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-10 12:41:08,515 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-16) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host rack is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-10 12:41:08,521 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-16) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues' > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:183) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:16) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor45.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] > >2014-09-10 12:41:11,545 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to dell-r210ii-16.rhev.lab.eng.brq.redhat.com/10.34.62.208 >2014-09-10 12:41:14,676 INFO [org.ovirt.engine.core.bll.storage.FenceVdsManualyCommand] (ajp-/127.0.0.1:8702-1) [59fa8473] Lock Acquired to object EngineLock [exclusiveLocks= key: 7ebdb254-efb6-480b-8c4e-a7dbdc759ed4 value: VDS_FENCE >, sharedLocks= ] >2014-09-10 12:41:14,677 INFO [org.ovirt.engine.core.bll.storage.FenceVdsManualyCommand] (org.ovirt.thread.pool-6-thread-35) [59fa8473] Running command: FenceVdsManualyCommand internal: false. Entities affected : ID: 7ebdb254-efb6-480b-8c4e-a7dbdc759ed4 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2014-09-10 12:41:14,686 INFO [org.ovirt.engine.core.bll.ClearNonResponsiveVdsVmsCommand] (org.ovirt.thread.pool-6-thread-35) [49d242c4] Running command: ClearNonResponsiveVdsVmsCommand internal: true. Entities affected : ID: 7ebdb254-efb6-480b-8c4e-a7dbdc759ed4 Type: VDS >2014-09-10 12:41:14,692 INFO [org.ovirt.engine.core.vdsbroker.UpdateVdsVMsClearedVDSCommand] (org.ovirt.thread.pool-6-thread-35) [49d242c4] START, UpdateVdsVMsClearedVDSCommand(HostName = rack, HostId = 7ebdb254-efb6-480b-8c4e-a7dbdc759ed4), log id: 23d936e7 >2014-09-10 12:41:28,434 INFO [org.ovirt.engine.core.bll.MaintenanceNumberOfVdssCommand] (org.ovirt.thread.pool-6-thread-36) [39b96113] Running command: MaintenanceNumberOfVdssCommand internal: false. Entities affected : ID: 7ebdb254-efb6-480b-8c4e-a7dbdc759ed4 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2014-09-10 12:41:28,481 INFO [org.ovirt.engine.core.bll.MaintenanceVdsCommand] (org.ovirt.thread.pool-6-thread-36) [39b96113] Running command: MaintenanceVdsCommand internal: true. Entities affected : ID: 7ebdb254-efb6-480b-8c4e-a7dbdc759ed4 Type: VDS >2014-09-10 12:41:28,487 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-6-thread-36) [39b96113] START, SetVdsStatusVDSCommand(HostName = rack, HostId = 7ebdb254-efb6-480b-8c4e-a7dbdc759ed4, status=Maintenance, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 1757e050 >2014-09-10 12:44:13,573 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-79) Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@91bf26 >2014-09-10 12:44:13,574 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-79) HostName = rack >2014-09-10 12:44:13,574 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-79) Command GetCapabilitiesVDSCommand(HostName = rack, HostId = 7ebdb254-efb6-480b-8c4e-a7dbdc759ed4, vds=Host[rack,7ebdb254-efb6-480b-8c4e-a7dbdc759ed4]) execution failed. Exception: VDSNetworkException: VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues' >2014-09-10 12:44:13,574 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-79) Host rack is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-10 12:44:13,588 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-79) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host rack is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-09-10 12:44:13,595 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-79) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues' > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:183) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:16) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:639) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:486) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor45.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] > >2014-09-10 12:44:13,631 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-6-thread-36) [39b96113] FINISH, SetVdsStatusVDSCommand, log id: 1757e050 >2014-09-10 12:44:13,646 INFO [org.ovirt.engine.core.vdsbroker.UpdateVdsVMsClearedVDSCommand] (org.ovirt.thread.pool-6-thread-35) [49d242c4] FINISH, UpdateVdsVMsClearedVDSCommand, log id: 23d936e7 >2014-09-10 12:44:13,655 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-36) [39b96113] Correlation ID: 39b96113, Job ID: 34b4e15b-e544-4bdc-afd3-6367282835af, Call Stack: null, Custom Event ID: -1, Message: Host rack was switched to Maintenance mode by admin. >2014-09-10 12:44:13,661 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-35) [49d242c4] Correlation ID: 49d242c4, Call Stack: null, Custom Event ID: -1, Message: All VMs' status on Non Responsive Host rack were changed to 'Down' by admin >2014-09-10 12:44:13,702 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-35) [49d242c4] Correlation ID: 59fa8473, Call Stack: null, Custom Event ID: -1, Message: Manual fence for host rack was started. >2014-09-10 12:44:13,703 INFO [org.ovirt.engine.core.bll.storage.FenceVdsManualyCommand] (org.ovirt.thread.pool-6-thread-35) [49d242c4] Lock freed to object EngineLock [exclusiveLocks= key: 7ebdb254-efb6-480b-8c4e-a7dbdc759ed4 value: VDS_FENCE >, sharedLocks= ] >2014-09-10 12:58:35,474 INFO [org.ovirt.engine.core.bll.MaintenanceNumberOfVdssCommand] (org.ovirt.thread.pool-6-thread-38) [3bff881a] Running command: MaintenanceNumberOfVdssCommand internal: false. Entities affected : ID: 7ebdb254-efb6-480b-8c4e-a7dbdc759ed4 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2014-09-10 12:58:35,527 INFO [org.ovirt.engine.core.bll.MaintenanceVdsCommand] (org.ovirt.thread.pool-6-thread-38) [3bff881a] Running command: MaintenanceVdsCommand internal: true. Entities affected : ID: 7ebdb254-efb6-480b-8c4e-a7dbdc759ed4 Type: VDS >2014-09-10 12:58:35,534 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-6-thread-38) [3bff881a] START, SetVdsStatusVDSCommand(HostName = rack, HostId = 7ebdb254-efb6-480b-8c4e-a7dbdc759ed4, status=Maintenance, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 5f143ac >2014-09-10 12:58:48,564 ERROR [org.ovirt.engine.core.utils.timer.SchedulerUtilQuartzImpl] (DefaultQuartzScheduler_Worker-44) Failed to invoke scheduled method refreshLightWeightData: java.lang.reflect.InvocationTargetException > at sun.reflect.GeneratedMethodAccessor61.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: org.springframework.jdbc.CannotGetJdbcConnectionException: Could not get JDBC Connection; nested exception is java.sql.SQLException: javax.resource.ResourceException: IJ000451: The connection manager is shutdown: java:/ENGINEDataSource > at org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:80) [spring-jdbc.jar:3.1.1.RELEASE] > at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:573) [spring-jdbc.jar:3.1.1.RELEASE] > at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:637) [spring-jdbc.jar:3.1.1.RELEASE] > at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:666) [spring-jdbc.jar:3.1.1.RELEASE] > at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:706) [spring-jdbc.jar:3.1.1.RELEASE] > at org.ovirt.engine.core.dal.dbbroker.PostgresDbEngineDialect$PostgresSimpleJdbcCall.executeCallInternal(PostgresDbEngineDialect.java:154) [dal.jar:] > at org.ovirt.engine.core.dal.dbbroker.PostgresDbEngineDialect$PostgresSimpleJdbcCall.doExecute(PostgresDbEngineDialect.java:120) [dal.jar:] > at org.springframework.jdbc.core.simple.SimpleJdbcCall.execute(SimpleJdbcCall.java:181) [spring-jdbc.jar:3.1.1.RELEASE] > at org.ovirt.engine.core.dal.dbbroker.SimpleJdbcCallsHandler.executeImpl(SimpleJdbcCallsHandler.java:141) [dal.jar:] > at org.ovirt.engine.core.dal.dbbroker.SimpleJdbcCallsHandler.executeReadList(SimpleJdbcCallsHandler.java:103) [dal.jar:] > at org.ovirt.engine.core.dao.VdsGroupDAODbFacadeImpl.getAll(VdsGroupDAODbFacadeImpl.java:135) [dal.jar:] > at org.ovirt.engine.core.dao.VdsGroupDAODbFacadeImpl.getAll(VdsGroupDAODbFacadeImpl.java:128) [dal.jar:] > at org.ovirt.engine.core.bll.gluster.GlusterSyncJob.refreshLightWeightData(GlusterSyncJob.java:82) [bll.jar:] > ... 6 more >Caused by: java.sql.SQLException: javax.resource.ResourceException: IJ000451: The connection manager is shutdown: java:/ENGINEDataSource > at org.jboss.jca.adapters.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:151) > at org.springframework.jdbc.datasource.DataSourceUtils.doGetConnection(DataSourceUtils.java:111) [spring-jdbc.jar:3.1.1.RELEASE] > at org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:77) [spring-jdbc.jar:3.1.1.RELEASE] > ... 18 more >Caused by: javax.resource.ResourceException: IJ000451: The connection manager is shutdown: java:/ENGINEDataSource > at org.jboss.jca.core.connectionmanager.AbstractConnectionManager.getManagedConnection(AbstractConnectionManager.java:345) > at org.jboss.jca.core.connectionmanager.tx.TxConnectionManagerImpl.getManagedConnection(TxConnectionManagerImpl.java:368) > at org.jboss.jca.core.connectionmanager.AbstractConnectionManager.allocateConnection(AbstractConnectionManager.java:488) > at org.jboss.jca.adapters.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:143) > ... 20 more > >2014-09-10 12:58:48,564 ERROR [org.ovirt.engine.core.utils.timer.SchedulerUtilQuartzImpl] (DefaultQuartzScheduler_Worker-46) Failed to invoke scheduled method refreshGlusterServices: java.lang.reflect.InvocationTargetException > at sun.reflect.GeneratedMethodAccessor62.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: org.springframework.jdbc.CannotGetJdbcConnectionException: Could not get JDBC Connection; nested exception is java.sql.SQLException: javax.resource.ResourceException: IJ000451: The connection manager is shutdown: java:/ENGINEDataSource > at org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:80) [spring-jdbc.jar:3.1.1.RELEASE] > at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:573) [spring-jdbc.jar:3.1.1.RELEASE] > at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:637) [spring-jdbc.jar:3.1.1.RELEASE] > at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:666) [spring-jdbc.jar:3.1.1.RELEASE] > at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:706) [spring-jdbc.jar:3.1.1.RELEASE] > at org.ovirt.engine.core.dal.dbbroker.PostgresDbEngineDialect$PostgresSimpleJdbcCall.executeCallInternal(PostgresDbEngineDialect.java:154) [dal.jar:] > at org.ovirt.engine.core.dal.dbbroker.PostgresDbEngineDialect$PostgresSimpleJdbcCall.doExecute(PostgresDbEngineDialect.java:120) [dal.jar:] > at org.springframework.jdbc.core.simple.SimpleJdbcCall.execute(SimpleJdbcCall.java:181) [spring-jdbc.jar:3.1.1.RELEASE] > at org.ovirt.engine.core.dal.dbbroker.SimpleJdbcCallsHandler.executeImpl(SimpleJdbcCallsHandler.java:141) [dal.jar:] > at org.ovirt.engine.core.dal.dbbroker.SimpleJdbcCallsHandler.executeReadList(SimpleJdbcCallsHandler.java:103) [dal.jar:] > at org.ovirt.engine.core.dao.VdsGroupDAODbFacadeImpl.getAll(VdsGroupDAODbFacadeImpl.java:135) [dal.jar:] > at org.ovirt.engine.core.dao.VdsGroupDAODbFacadeImpl.getAll(VdsGroupDAODbFacadeImpl.java:128) [dal.jar:] > at org.ovirt.engine.core.bll.gluster.GlusterServiceSyncJob.refreshGlusterServices(GlusterServiceSyncJob.java:54) [bll.jar:] > ... 6 more >Caused by: java.sql.SQLException: javax.resource.ResourceException: IJ000451: The connection manager is shutdown: java:/ENGINEDataSource > at org.jboss.jca.adapters.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:151) > at org.springframework.jdbc.datasource.DataSourceUtils.doGetConnection(DataSourceUtils.java:111) [spring-jdbc.jar:3.1.1.RELEASE] > at org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:77) [spring-jdbc.jar:3.1.1.RELEASE] > ... 18 more >Caused by: javax.resource.ResourceException: IJ000451: The connection manager is shutdown: java:/ENGINEDataSource > at org.jboss.jca.core.connectionmanager.AbstractConnectionManager.getManagedConnection(AbstractConnectionManager.java:345) > at org.jboss.jca.core.connectionmanager.tx.TxConnectionManagerImpl.getManagedConnection(TxConnectionManagerImpl.java:368) > at org.jboss.jca.core.connectionmanager.AbstractConnectionManager.allocateConnection(AbstractConnectionManager.java:488) > at org.jboss.jca.adapters.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:143) > ... 20 more > >2014-09-10 12:58:53,574 ERROR [org.ovirt.engine.core.utils.timer.SchedulerUtilQuartzImpl] (DefaultQuartzScheduler_Worker-59) Failed to invoke scheduled method refreshLightWeightData: java.lang.reflect.InvocationTargetException > at sun.reflect.GeneratedMethodAccessor61.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: org.springframework.jdbc.CannotGetJdbcConnectionException: Could not get JDBC Connection; nested exception is java.sql.SQLException: javax.resource.ResourceException: IJ000451: The connection manager is shutdown: java:/ENGINEDataSource > at org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:80) [spring-jdbc.jar:3.1.1.RELEASE] > at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:573) [spring-jdbc.jar:3.1.1.RELEASE] > at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:637) [spring-jdbc.jar:3.1.1.RELEASE] > at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:666) [spring-jdbc.jar:3.1.1.RELEASE] > at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:706) [spring-jdbc.jar:3.1.1.RELEASE] > at org.ovirt.engine.core.dal.dbbroker.PostgresDbEngineDialect$PostgresSimpleJdbcCall.executeCallInternal(PostgresDbEngineDialect.java:154) [dal.jar:] > at org.ovirt.engine.core.dal.dbbroker.PostgresDbEngineDialect$PostgresSimpleJdbcCall.doExecute(PostgresDbEngineDialect.java:120) [dal.jar:] > at org.springframework.jdbc.core.simple.SimpleJdbcCall.execute(SimpleJdbcCall.java:181) [spring-jdbc.jar:3.1.1.RELEASE] > at org.ovirt.engine.core.dal.dbbroker.SimpleJdbcCallsHandler.executeImpl(SimpleJdbcCallsHandler.java:141) [dal.jar:] > at org.ovirt.engine.core.dal.dbbroker.SimpleJdbcCallsHandler.executeReadList(SimpleJdbcCallsHandler.java:103) [dal.jar:] > at org.ovirt.engine.core.dao.VdsGroupDAODbFacadeImpl.getAll(VdsGroupDAODbFacadeImpl.java:135) [dal.jar:] > at org.ovirt.engine.core.dao.VdsGroupDAODbFacadeImpl.getAll(VdsGroupDAODbFacadeImpl.java:128) [dal.jar:] > at org.ovirt.engine.core.bll.gluster.GlusterSyncJob.refreshLightWeightData(GlusterSyncJob.java:82) [bll.jar:] > ... 6 more >Caused by: java.sql.SQLException: javax.resource.ResourceException: IJ000451: The connection manager is shutdown: java:/ENGINEDataSource > at org.jboss.jca.adapters.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:151) > at org.springframework.jdbc.datasource.DataSourceUtils.doGetConnection(DataSourceUtils.java:111) [spring-jdbc.jar:3.1.1.RELEASE] > at org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:77) [spring-jdbc.jar:3.1.1.RELEASE] > ... 18 more >Caused by: javax.resource.ResourceException: IJ000451: The connection manager is shutdown: java:/ENGINEDataSource > at org.jboss.jca.core.connectionmanager.AbstractConnectionManager.getManagedConnection(AbstractConnectionManager.java:345) > at org.jboss.jca.core.connectionmanager.tx.TxConnectionManagerImpl.getManagedConnection(TxConnectionManagerImpl.java:368) > at org.jboss.jca.core.connectionmanager.AbstractConnectionManager.allocateConnection(AbstractConnectionManager.java:488) > at org.jboss.jca.adapters.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:143) > ... 20 more > >2014-09-10 12:58:53,573 ERROR [org.ovirt.engine.core.utils.timer.SchedulerUtilQuartzImpl] (DefaultQuartzScheduler_Worker-56) Failed to invoke scheduled method refreshGlusterServices: java.lang.reflect.InvocationTargetException > at sun.reflect.GeneratedMethodAccessor62.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: org.springframework.jdbc.CannotGetJdbcConnectionException: Could not get JDBC Connection; nested exception is java.sql.SQLException: javax.resource.ResourceException: IJ000451: The connection manager is shutdown: java:/ENGINEDataSource > at org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:80) [spring-jdbc.jar:3.1.1.RELEASE] > at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:573) [spring-jdbc.jar:3.1.1.RELEASE] > at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:637) [spring-jdbc.jar:3.1.1.RELEASE] > at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:666) [spring-jdbc.jar:3.1.1.RELEASE] > at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:706) [spring-jdbc.jar:3.1.1.RELEASE] > at org.ovirt.engine.core.dal.dbbroker.PostgresDbEngineDialect$PostgresSimpleJdbcCall.executeCallInternal(PostgresDbEngineDialect.java:154) [dal.jar:] > at org.ovirt.engine.core.dal.dbbroker.PostgresDbEngineDialect$PostgresSimpleJdbcCall.doExecute(PostgresDbEngineDialect.java:120) [dal.jar:] > at org.springframework.jdbc.core.simple.SimpleJdbcCall.execute(SimpleJdbcCall.java:181) [spring-jdbc.jar:3.1.1.RELEASE] > at org.ovirt.engine.core.dal.dbbroker.SimpleJdbcCallsHandler.executeImpl(SimpleJdbcCallsHandler.java:141) [dal.jar:] > at org.ovirt.engine.core.dal.dbbroker.SimpleJdbcCallsHandler.executeReadList(SimpleJdbcCallsHandler.java:103) [dal.jar:] > at org.ovirt.engine.core.dao.VdsGroupDAODbFacadeImpl.getAll(VdsGroupDAODbFacadeImpl.java:135) [dal.jar:] > at org.ovirt.engine.core.dao.VdsGroupDAODbFacadeImpl.getAll(VdsGroupDAODbFacadeImpl.java:128) [dal.jar:] > at org.ovirt.engine.core.bll.gluster.GlusterServiceSyncJob.refreshGlusterServices(GlusterServiceSyncJob.java:54) [bll.jar:] > ... 6 more >Caused by: java.sql.SQLException: javax.resource.ResourceException: IJ000451: The connection manager is shutdown: java:/ENGINEDataSource > at org.jboss.jca.adapters.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:151) > at org.springframework.jdbc.datasource.DataSourceUtils.doGetConnection(DataSourceUtils.java:111) [spring-jdbc.jar:3.1.1.RELEASE] > at org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:77) [spring-jdbc.jar:3.1.1.RELEASE] > ... 18 more >Caused by: javax.resource.ResourceException: IJ000451: The connection manager is shutdown: java:/ENGINEDataSource > at org.jboss.jca.core.connectionmanager.AbstractConnectionManager.getManagedConnection(AbstractConnectionManager.java:345) > at org.jboss.jca.core.connectionmanager.tx.TxConnectionManagerImpl.getManagedConnection(TxConnectionManagerImpl.java:368) > at org.jboss.jca.core.connectionmanager.AbstractConnectionManager.allocateConnection(AbstractConnectionManager.java:488) > at org.jboss.jca.adapters.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:143) > ... 20 more > >2014-09-10 12:59:04,519 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Loaded file "/usr/share/ovirt-engine/services/ovirt-engine/ovirt-engine.conf". >2014-09-10 12:59:04,519 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) The file "/etc/ovirt-engine/engine.conf" doesn't exist or isn't readable. Will return an empty set of properties. >2014-09-10 12:59:04,519 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Loaded file "/etc/ovirt-engine/engine.conf.d/10-rhevm-doc.conf". >2014-09-10 12:59:04,520 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Loaded file "/etc/ovirt-engine/engine.conf.d/10-setup-database.conf". >2014-09-10 12:59:04,520 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Loaded file "/etc/ovirt-engine/engine.conf.d/10-setup-jboss.conf". >2014-09-10 12:59:04,520 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Loaded file "/etc/ovirt-engine/engine.conf.d/10-setup-pki.conf". >2014-09-10 12:59:04,520 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Loaded file "/etc/ovirt-engine/engine.conf.d/10-setup-protocols.conf". >2014-09-10 12:59:04,521 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Loaded file "/etc/ovirt-engine/engine.conf.d/20-branding-rhev.conf". >2014-09-10 12:59:04,521 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Loaded file "/etc/ovirt-engine/engine.conf.d/50-redhat-support-plugin.conf". >2014-09-10 12:59:04,521 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Value of property "ENGINE_AJP_ENABLED" is "true". >2014-09-10 12:59:04,521 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Value of property "ENGINE_AJP_PORT" is "8702". >2014-09-10 12:59:04,521 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Value of property "ENGINE_APPS" is "engine.ear restapi.war legacy_restapi.war "/usr/share/ovirt-engine/branding/rhev-2.brand/applications/rhev.ear" redhat-support-plugin-rhev.ear". >2014-09-10 12:59:04,521 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Value of property "ENGINE_CACHE" is "/var/cache/ovirt-engine". >2014-09-10 12:59:04,522 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Value of property "ENGINE_DB_CHECK_INTERVAL" is "1000". >2014-09-10 12:59:04,522 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Value of property "ENGINE_DB_CONNECTION_TIMEOUT" is "300000". >2014-09-10 12:59:04,522 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Value of property "ENGINE_DB_DATABASE" is "engine". >2014-09-10 12:59:04,522 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Value of property "ENGINE_DB_DRIVER" is "org.postgresql.Driver". >2014-09-10 12:59:04,522 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Value of property "ENGINE_DB_HOST" is "localhost". >2014-09-10 12:59:04,522 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Value of property "ENGINE_DB_MAX_CONNECTIONS" is "100". >2014-09-10 12:59:04,522 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Value of property "ENGINE_DB_MIN_CONNECTIONS" is "1". >2014-09-10 12:59:04,522 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Value of property "ENGINE_DB_PASSWORD" is "***". >2014-09-10 12:59:04,523 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Value of property "ENGINE_DB_PORT" is "5432". >2014-09-10 12:59:04,523 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Value of property "ENGINE_DB_SECURED" is "False". >2014-09-10 12:59:04,523 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Value of property "ENGINE_DB_SECURED_VALIDATION" is "False". >2014-09-10 12:59:04,523 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Value of property "ENGINE_DB_URL" is "jdbc:postgresql://localhost:5432/engine?sslfactory=org.postgresql.ssl.NonValidatingFactory". >2014-09-10 12:59:04,523 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Value of property "ENGINE_DB_USER" is "engine". >2014-09-10 12:59:04,523 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Value of property "ENGINE_DEBUG_ADDRESS" is "". >2014-09-10 12:59:04,523 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Value of property "ENGINE_DOC" is "/usr/share/doc/ovirt-engine". >2014-09-10 12:59:04,531 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Value of property "ENGINE_ETC" is "/etc/ovirt-engine". >2014-09-10 12:59:04,532 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Value of property "ENGINE_EXTENSION_PATH" is "/usr/share/ovirt-engine/extensions.d:/etc/ovirt-engine/extensions.d". >2014-09-10 12:59:04,532 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Value of property "ENGINE_FQDN" is "pb-rh35.rhev.lab.eng.brq.redhat.com". >2014-09-10 12:59:04,532 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Value of property "ENGINE_GROUP" is "ovirt". >2014-09-10 12:59:04,532 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Value of property "ENGINE_HEAP_MAX" is "1g". >2014-09-10 12:59:04,532 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Value of property "ENGINE_HEAP_MIN" is "1g". >2014-09-10 12:59:04,532 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Value of property "ENGINE_HTTPS_ENABLED" is "false". >2014-09-10 12:59:04,532 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Value of property "ENGINE_HTTPS_PORT" is "None". >2014-09-10 12:59:04,532 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Value of property "ENGINE_HTTPS_PROTOCOLS" is "SSLv3,TLSv1,TLSv1.1,TLSv1.2". >2014-09-10 12:59:04,532 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Value of property "ENGINE_HTTP_ENABLED" is "false". >2014-09-10 12:59:04,532 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Value of property "ENGINE_HTTP_PORT" is "None". >2014-09-10 12:59:04,533 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Value of property "ENGINE_JAVA_MODULEPATH" is "/usr/share/ovirt-engine/modules". >2014-09-10 12:59:04,533 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Value of property "ENGINE_JVM_ARGS" is " -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath="/var/log/ovirt-engine/dump"". >2014-09-10 12:59:04,533 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Value of property "ENGINE_LOG" is "/var/log/ovirt-engine". >2014-09-10 12:59:04,533 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Value of property "ENGINE_LOG_TO_CONSOLE" is "false". >2014-09-10 12:59:04,533 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Value of property "ENGINE_MANUAL" is "/usr/share/doc/rhevm-doc/manual". >2014-09-10 12:59:04,533 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Value of property "ENGINE_PERM_MAX" is "256m". >2014-09-10 12:59:04,533 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Value of property "ENGINE_PERM_MIN" is "256m". >2014-09-10 12:59:04,533 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Value of property "ENGINE_PKI" is "/etc/pki/ovirt-engine". >2014-09-10 12:59:04,534 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Value of property "ENGINE_PKI_CA" is "/etc/pki/ovirt-engine/ca.pem". >2014-09-10 12:59:04,534 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Value of property "ENGINE_PKI_ENGINE_CERT" is "/etc/pki/ovirt-engine/certs/engine.cer". >2014-09-10 12:59:04,534 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Value of property "ENGINE_PKI_ENGINE_STORE" is "/etc/pki/ovirt-engine/keys/engine.p12". >2014-09-10 12:59:04,534 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Value of property "ENGINE_PKI_ENGINE_STORE_ALIAS" is "1". >2014-09-10 12:59:04,534 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Value of property "ENGINE_PKI_ENGINE_STORE_PASSWORD" is "***". >2014-09-10 12:59:04,534 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Value of property "ENGINE_PKI_TRUST_STORE" is "/etc/pki/ovirt-engine/.truststore". >2014-09-10 12:59:04,534 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Value of property "ENGINE_PKI_TRUST_STORE_PASSWORD" is "***". >2014-09-10 12:59:04,534 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Value of property "ENGINE_PROPERTIES" is " jsse.enableSNIExtension=false "java.security.krb5.conf=/etc/ovirt-engine/krb5.conf"". >2014-09-10 12:59:04,534 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Value of property "ENGINE_PROXY_ENABLED" is "true". >2014-09-10 12:59:04,534 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Value of property "ENGINE_PROXY_HTTPS_PORT" is "443". >2014-09-10 12:59:04,535 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Value of property "ENGINE_PROXY_HTTP_PORT" is "80". >2014-09-10 12:59:04,535 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Value of property "ENGINE_REPORTS_UI" is "/var/lib/ovirt-engine/reports.xml". >2014-09-10 12:59:04,535 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Value of property "ENGINE_STOP_INTERVAL" is "1". >2014-09-10 12:59:04,535 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Value of property "ENGINE_STOP_TIME" is "10". >2014-09-10 12:59:04,535 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Value of property "ENGINE_TMP" is "/var/tmp/ovirt-engine". >2014-09-10 12:59:04,535 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Value of property "ENGINE_UP_MARK" is "/var/lib/ovirt-engine/engine.up". >2014-09-10 12:59:04,543 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Value of property "ENGINE_URI" is "/ovirt-engine". >2014-09-10 12:59:04,544 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Value of property "ENGINE_USER" is "ovirt". >2014-09-10 12:59:04,544 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Value of property "ENGINE_USR" is "/usr/share/ovirt-engine". >2014-09-10 12:59:04,544 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Value of property "ENGINE_VAR" is "/var/lib/ovirt-engine". >2014-09-10 12:59:04,544 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Value of property "ENGINE_VERBOSE_GC" is "false". >2014-09-10 12:59:04,544 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Value of property "JBOSS_HOME" is "/usr/share/jbossas". >2014-09-10 12:59:04,544 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Value of property "JBOSS_RUNTIME" is "/var/lib/ovirt-engine/jboss_runtime". >2014-09-10 12:59:04,544 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 38) Value of property "SENSITIVE_KEYS" is ",ENGINE_DB_PASSWORD,ENGINE_PKI_TRUST_STORE_PASSWORD,ENGINE_PKI_ENGINE_STORE_PASSWORD". >2014-09-10 12:59:13,281 INFO [org.ovirt.engine.core.bll.Backend] (ServerService Thread Pool -- 49) Start initializing Backend >2014-09-10 12:59:13,790 INFO [org.ovirt.engine.core.utils.osinfo.OsInfoPreferencesLoader] (ServerService Thread Pool -- 49) Loaded file /etc/ovirt-engine/osinfo.conf.d/00-defaults.properties >2014-09-10 12:59:13,933 INFO [org.ovirt.engine.core.bll.Backend] (ServerService Thread Pool -- 49) Running ovirt-engine 3.5.0-0.10.master.el6ev >2014-09-10 12:59:13,933 INFO [org.ovirt.engine.core.bll.CpuFlagsManagerHandler] (ServerService Thread Pool -- 49) Start initializing dictionaries >2014-09-10 12:59:13,935 INFO [org.ovirt.engine.core.bll.CpuFlagsManagerHandler] (ServerService Thread Pool -- 49) Finished initializing dictionaries >2014-09-10 12:59:13,935 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (ServerService Thread Pool -- 49) Start initializing AuditLogCleanupManager >2014-09-10 12:59:13,935 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (ServerService Thread Pool -- 49) Setting audit cleanup manager to run at: 35 35 3 * * ? >2014-09-10 12:59:13,958 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (ServerService Thread Pool -- 49) Finished initializing AuditLogCleanupManager >2014-09-10 12:59:13,959 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (ServerService Thread Pool -- 49) Start initializing CommandEntityCleanupManager >2014-09-10 12:59:13,959 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (ServerService Thread Pool -- 49) Setting command entity cleanup manager to run at: 35 35 3 * * ? >2014-09-10 12:59:13,961 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (ServerService Thread Pool -- 49) Finished initializing CommandEntityCleanupManager >2014-09-10 12:59:13,963 INFO [org.ovirt.engine.core.bll.TagsDirector] (ServerService Thread Pool -- 49) Start initializing TagsDirector >2014-09-10 12:59:13,973 INFO [org.ovirt.engine.core.bll.TagsDirector] (ServerService Thread Pool -- 49) Tag root added to tree >2014-09-10 12:59:13,990 INFO [org.ovirt.engine.core.bll.TagsDirector] (ServerService Thread Pool -- 49) Finished initializing TagsDirector >2014-09-10 12:59:13,990 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (ServerService Thread Pool -- 49) Start initializing IsoDomainListSyncronizer >2014-09-10 12:59:13,996 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (ServerService Thread Pool -- 49) Finished initializing IsoDomainListSyncronizer >2014-09-10 12:59:14,032 INFO [org.ovirt.engine.core.bll.Backend] (ServerService Thread Pool -- 49) Completed initializing handlers >2014-09-10 12:59:14,060 INFO [org.ovirt.engine.core.utils.ErrorTranslatorImpl] (ServerService Thread Pool -- 49) Start initializing ErrorTranslatorImpl >2014-09-10 12:59:14,073 WARN [org.ovirt.engine.core.utils.ErrorTranslatorImpl] (ServerService Thread Pool -- 49) Code MAC_ADDRESS_IS_IN_USE appears more than once in string table. >2014-09-10 12:59:14,074 INFO [org.ovirt.engine.core.utils.ErrorTranslatorImpl] (ServerService Thread Pool -- 49) Finished initializing ErrorTranslatorImpl >2014-09-10 12:59:14,075 INFO [org.ovirt.engine.core.utils.ErrorTranslatorImpl] (ServerService Thread Pool -- 49) Start initializing ErrorTranslatorImpl >2014-09-10 12:59:14,075 INFO [org.ovirt.engine.core.utils.ErrorTranslatorImpl] (ServerService Thread Pool -- 49) Finished initializing ErrorTranslatorImpl >2014-09-10 12:59:14,075 INFO [org.ovirt.engine.core.bll.Backend] (ServerService Thread Pool -- 49) Mark incomplete jobs as UNKNOWN >2014-09-10 12:59:14,121 INFO [org.ovirt.engine.core.bll.job.JobRepositoryCleanupManager] (ServerService Thread Pool -- 49) Start initializing JobRepositoryCleanupManager >2014-09-10 12:59:14,121 INFO [org.ovirt.engine.core.bll.job.JobRepositoryCleanupManager] (ServerService Thread Pool -- 49) Finished initializing JobRepositoryCleanupManager >2014-09-10 12:59:14,122 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (ServerService Thread Pool -- 49) Start initializing AutoRecoveryManager >2014-09-10 12:59:14,123 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (ServerService Thread Pool -- 49) Finished initializing AutoRecoveryManager >2014-09-10 12:59:14,125 INFO [org.ovirt.engine.core.dal.job.ExecutionMessageDirector] (ServerService Thread Pool -- 49) Start initializing ExecutionMessageDirector >2014-09-10 12:59:14,129 INFO [org.ovirt.engine.core.dal.job.ExecutionMessageDirector] (ServerService Thread Pool -- 49) Finished initializing ExecutionMessageDirector >2014-09-10 12:59:14,252 INFO [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (ServerService Thread Pool -- 49) Loading extension 'builtin-authn-internal' >2014-09-10 12:59:14,253 INFO [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (ServerService Thread Pool -- 49) Extension 'builtin-authn-internal' loaded >2014-09-10 12:59:14,255 INFO [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (ServerService Thread Pool -- 49) Loading extension 'internal' >2014-09-10 12:59:14,257 INFO [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (ServerService Thread Pool -- 49) Extension 'internal' loaded >2014-09-10 12:59:14,257 INFO [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (ServerService Thread Pool -- 49) Initializing extension 'builtin-authn-internal' >2014-09-10 12:59:14,257 INFO [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (ServerService Thread Pool -- 49) Extension 'builtin-authn-internal' initialized >2014-09-10 12:59:14,258 INFO [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (ServerService Thread Pool -- 49) Initializing extension 'internal' >2014-09-10 12:59:14,258 INFO [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (ServerService Thread Pool -- 49) Extension 'internal' initialized >2014-09-10 12:59:14,258 INFO [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (ServerService Thread Pool -- 49) Start of enabled extensions list >2014-09-10 12:59:14,258 INFO [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (ServerService Thread Pool -- 49) Instance name: 'builtin-authn-internal', Extension name: 'Internal Authn (Built-in)', Version: 'N/A', Notes: '', License: 'ASL 2.0', Home: 'http://www.ovirt.org', Author 'The oVirt Project', Build interface Version: '0', File: 'N/A', Initialized: 'true' >2014-09-10 12:59:14,258 INFO [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (ServerService Thread Pool -- 49) Instance name: 'internal', Extension name: 'Internal Authz (Built-in)', Version: 'N/A', Notes: '', License: 'ASL 2.0', Home: 'http://www.ovirt.org', Author 'The oVirt Project', Build interface Version: '0', File: 'N/A', Initialized: 'true' >2014-09-10 12:59:14,258 INFO [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (ServerService Thread Pool -- 49) End of enabled extensions list >2014-09-10 12:59:14,434 INFO [org.ovirt.engine.core.bll.aaa.DbUserCacheManager] (ServerService Thread Pool -- 54) Start initializing DbUserCacheManager >2014-09-10 12:59:14,446 INFO [org.ovirt.engine.core.bll.aaa.DbUserCacheManager] (ServerService Thread Pool -- 54) Finished initializing DbUserCacheManager >2014-09-10 12:59:14,496 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (ServerService Thread Pool -- 54) Initialization of AsyncTaskManager completed successfully. >2014-09-10 12:59:14,498 INFO [org.ovirt.engine.core.vdsbroker.ResourceManager] (ServerService Thread Pool -- 54) Start initializing ResourceManager >2014-09-10 12:59:14,599 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-1) Principal admin::internal synchronized >2014-09-10 12:59:14,817 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (ServerService Thread Pool -- 54) Entered VdsManager constructor >2014-09-10 12:59:14,921 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (ServerService Thread Pool -- 54) Initialize vdsBroker (dell-r210ii-16.rhev.lab.eng.brq.redhat.com,54,321) >2014-09-10 12:59:15,118 INFO [org.ovirt.engine.core.vdsbroker.ResourceManager] (ServerService Thread Pool -- 54) VDS 7ebdb254-efb6-480b-8c4e-a7dbdc759ed4 was added to the Resource Manager >2014-09-10 12:59:15,171 INFO [org.ovirt.engine.core.vdsbroker.ResourceManager] (ServerService Thread Pool -- 54) Finished initializing ResourceManager >2014-09-10 12:59:15,178 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (ServerService Thread Pool -- 54) Initialization of OvfDataUpdater completed successfully. >2014-09-10 12:59:15,179 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (ServerService Thread Pool -- 54) Start scheduling to enable vds load balancer >2014-09-10 12:59:15,179 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (ServerService Thread Pool -- 54) Finished scheduling to enable vds load balancer >2014-09-10 12:59:15,179 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (ServerService Thread Pool -- 54) Start HA Reservation check >2014-09-10 12:59:15,181 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (ServerService Thread Pool -- 54) Finished HA Reservation check >2014-09-10 12:59:15,186 INFO [org.ovirt.engine.core.bll.network.MacPoolManagerRanges] (org.ovirt.thread.pool-6-thread-1) Start initializing MacPoolManagerRanges >2014-09-10 12:59:15,201 INFO [org.ovirt.engine.core.bll.InitBackendServicesOnStartupBean] (ServerService Thread Pool -- 54) Init VM custom properties utilities >2014-09-10 12:59:15,202 INFO [org.ovirt.engine.core.bll.InitBackendServicesOnStartupBean] (ServerService Thread Pool -- 54) Init device custom properties utilities >2014-09-10 12:59:15,213 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (ServerService Thread Pool -- 54) Initializing Scheduling manager >2014-09-10 12:59:15,253 INFO [org.ovirt.engine.core.bll.network.MacPoolManagerRanges] (org.ovirt.thread.pool-6-thread-1) Finished initializing. Available MACs in pool: 65 >2014-09-10 12:59:15,297 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (ServerService Thread Pool -- 54) External scheduler disabled, discovery skipped >2014-09-10 12:59:15,297 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (ServerService Thread Pool -- 54) Initialized Scheduling manager >2014-09-10 12:59:15,298 INFO [org.ovirt.engine.core.bll.dwh.DwhHeartBeat] (ServerService Thread Pool -- 54) Initializing DWH Heart Beat >2014-09-10 12:59:15,299 INFO [org.ovirt.engine.core.bll.dwh.DwhHeartBeat] (ServerService Thread Pool -- 54) DWH Heart Beat initialized >2014-09-10 12:59:18,360 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to dell-r210ii-16.rhev.lab.eng.brq.redhat.com/10.34.62.208 >2014-09-10 12:59:18,922 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-6) START, GetHardwareInfoVDSCommand(HostName = rack, HostId = 7ebdb254-efb6-480b-8c4e-a7dbdc759ed4, vds=Host[rack,7ebdb254-efb6-480b-8c4e-a7dbdc759ed4]), log id: 737412bd >2014-09-10 12:59:19,004 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-6) FINISH, GetHardwareInfoVDSCommand, log id: 737412bd >2014-09-10 12:59:19,467 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-6) [34014e32] AuditLogType: UNASSIGNED not exist in string table >2014-09-10 12:59:19,468 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-6) [34014e32] AuditLogType: VDS_AUTO_FENCE_STATUS not exist in string table >2014-09-10 12:59:19,468 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-6) [34014e32] AuditLogType: VDS_AUTO_FENCE_STATUS_FAILED not exist in string table >2014-09-10 12:59:19,468 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-6) [34014e32] AuditLogType: VDS_AUTO_FENCE_FAILED_CALL_FENCE_SPM not exist in string table >2014-09-10 12:59:19,468 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-6) [34014e32] AuditLogType: VDS_HIGH_NETWORK_USE not exist in string table >2014-09-10 12:59:19,468 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-6) [34014e32] AuditLogType: USER_FAILED_REMOVE_VM not exist in string table >2014-09-10 12:59:19,469 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-6) [34014e32] AuditLogType: USER_RUN_UNLOCK_ENTITY_SCRIPT not exist in string table >2014-09-10 12:59:19,469 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-6) [34014e32] AuditLogType: VDS_NETWORK_MTU_DIFFER_FROM_LOGICAL_NETWORK not exist in string table >2014-09-10 12:59:19,470 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-6) [34014e32] AuditLogType: STORAGE_ACTIVATE_ASYNC not exist in string table >2014-09-10 12:59:19,470 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-6) [34014e32] AuditLogType: DWH_STOPPED not exist in string table >2014-09-10 12:59:19,470 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-6) [34014e32] AuditLogType: DWH_STARTED not exist in string table >2014-09-10 12:59:19,470 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-6) [34014e32] AuditLogType: DWH_ERROR not exist in string table >2014-09-10 12:59:19,470 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-6) [34014e32] AuditLogType: USER_ADDED_DISK_PROFILE not exist in string table >2014-09-10 12:59:19,470 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-6) [34014e32] AuditLogType: USER_FAILED_TO_ADD_DISK_PROFILE not exist in string table >2014-09-10 12:59:19,470 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-6) [34014e32] AuditLogType: USER_REMOVED_DISK_PROFILE not exist in string table >2014-09-10 12:59:19,471 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-6) [34014e32] AuditLogType: USER_FAILED_TO_REMOVE_DISK_PROFILE not exist in string table >2014-09-10 12:59:19,471 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-6) [34014e32] AuditLogType: USER_UPDATED_DISK_PROFILE not exist in string table >2014-09-10 12:59:19,471 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-6) [34014e32] AuditLogType: USER_FAILED_TO_UPDATE_DISK_PROFILE not exist in string table >2014-09-10 12:59:19,471 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-6) [34014e32] AuditLogType: USER_ADDED_CPU_PROFILE not exist in string table >2014-09-10 12:59:19,471 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-6) [34014e32] AuditLogType: USER_FAILED_TO_ADD_CPU_PROFILE not exist in string table >2014-09-10 12:59:19,471 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-6) [34014e32] AuditLogType: USER_REMOVED_CPU_PROFILE not exist in string table >2014-09-10 12:59:19,472 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-6) [34014e32] AuditLogType: USER_FAILED_TO_REMOVE_CPU_PROFILE not exist in string table >2014-09-10 12:59:19,472 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-6) [34014e32] AuditLogType: USER_UPDATED_CPU_PROFILE not exist in string table >2014-09-10 12:59:19,472 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-6) [34014e32] AuditLogType: USER_FAILED_TO_UPDATE_CPU_PROFILE not exist in string table >2014-09-10 12:59:19,472 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-6) [34014e32] AuditLogType: USER_REMOVE_AUDIT_LOG not exist in string table >2014-09-10 12:59:19,472 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-6) [34014e32] AuditLogType: USER_REMOVE_AUDIT_LOG_FAILED not exist in string table >2014-09-10 12:59:19,472 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-6) [34014e32] AuditLogType: USER_CLEAR_ALL_DISMISSED_AUDIT_LOG not exist in string table >2014-09-10 12:59:19,472 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-6) [34014e32] AuditLogType: USER_CLEAR_ALL_DISMISSED_AUDIT_LOG_FAILED not exist in string table >2014-09-10 12:59:19,807 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-6) [34014e32] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 7ebdb254-efb6-480b-8c4e-a7dbdc759ed4 Type: VDS >2014-09-10 12:59:19,814 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-6) [34014e32] START, SetVdsStatusVDSCommand(HostName = rack, HostId = 7ebdb254-efb6-480b-8c4e-a7dbdc759ed4, status=NonOperational, nonOperationalReason=NETWORK_UNREACHABLE, stopSpmFailureLogged=false), log id: 731862af >2014-09-10 12:59:19,862 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-6) [34014e32] FINISH, SetVdsStatusVDSCommand, log id: 731862af >2014-09-10 12:59:19,926 ERROR [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-6) [34014e32] Host rack is set to Non-Operational, it is missing the following networks: rhevm >2014-09-10 12:59:20,481 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-6) [34014e32] Correlation ID: 34014e32, Job ID: 7c21bda2-6400-41ce-a95f-e545b60e9006, Call Stack: null, Custom Event ID: -1, Message: Host rack does not comply with the cluster Default networks, the following networks are missing on host: 'rhevm' >2014-09-10 12:59:20,531 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-6) [7041b420] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 7ebdb254-efb6-480b-8c4e-a7dbdc759ed4 Type: VDS >2014-09-10 12:59:20,605 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-6) [7041b420] START, GetHardwareInfoVDSCommand(HostName = rack, HostId = 7ebdb254-efb6-480b-8c4e-a7dbdc759ed4, vds=Host[rack,7ebdb254-efb6-480b-8c4e-a7dbdc759ed4]), log id: 2698add4 >2014-09-10 12:59:20,610 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-6) [7041b420] FINISH, GetHardwareInfoVDSCommand, log id: 2698add4 >2014-09-10 12:59:20,933 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-6) [6acc6dc] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 7ebdb254-efb6-480b-8c4e-a7dbdc759ed4 Type: VDS >2014-09-10 12:59:20,938 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-6) [6acc6dc] START, SetVdsStatusVDSCommand(HostName = rack, HostId = 7ebdb254-efb6-480b-8c4e-a7dbdc759ed4, status=NonOperational, nonOperationalReason=NETWORK_UNREACHABLE, stopSpmFailureLogged=false), log id: 6e948eb3 >2014-09-10 12:59:20,961 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-6) [6acc6dc] FINISH, SetVdsStatusVDSCommand, log id: 6e948eb3 >2014-09-10 12:59:20,969 ERROR [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-6) [6acc6dc] Host rack is set to Non-Operational, it is missing the following networks: rhevm >2014-09-10 12:59:20,998 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-6) [16ec4930] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 7ebdb254-efb6-480b-8c4e-a7dbdc759ed4 Type: VDS >2014-09-10 12:59:21,022 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-6) [16ec4930] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host rack was set to NonOperational. >2014-09-10 12:59:21,276 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-6) [16ec4930] Initializing Host: rack >2014-09-10 12:59:21,302 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-6) [1a0445c6] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 7ebdb254-efb6-480b-8c4e-a7dbdc759ed4 Type: VDS >2014-09-10 12:59:21,316 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-6) [54660788] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 7ebdb254-efb6-480b-8c4e-a7dbdc759ed4 Type: VDS >2014-09-10 12:59:21,327 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-6) [54660788] Host 7ebdb254-efb6-480b-8c4e-a7dbdc759ed4 : rack is already in NonOperational status for reason NETWORK_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-09-10 12:59:33,380 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp-/127.0.0.1:8702-1) Running command: LoginUserCommand internal: false. >2014-09-10 12:59:33,406 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-1) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-09-10 12:59:40,227 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp-/127.0.0.1:8702-8) [51667022] Running command: LogoutUserCommand internal: false. >2014-09-10 12:59:40,238 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp-/127.0.0.1:8702-7) [26d7db08] Running command: LogoutUserCommand internal: false. >2014-09-10 12:59:40,240 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp-/127.0.0.1:8702-6) [23cc4a01] Running command: LogoutUserCommand internal: false. >2014-09-10 12:59:40,243 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp-/127.0.0.1:8702-5) [20c626ed] Running command: LogoutUserCommand internal: false. >2014-09-10 12:59:40,265 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp-/127.0.0.1:8702-4) [31c37a79] Running command: LogoutUserCommand internal: false. >2014-09-10 12:59:40,327 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp-/127.0.0.1:8702-3) [63537fc9] Running command: LogoutUserCommand internal: false. >2014-09-10 12:59:40,523 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-7) [26d7db08] Correlation ID: 26d7db08, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-09-10 12:59:40,524 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-8) [51667022] Correlation ID: 51667022, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-09-10 12:59:40,548 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-6) [23cc4a01] Correlation ID: 23cc4a01, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-09-10 12:59:40,560 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-5) [20c626ed] Correlation ID: 20c626ed, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-09-10 12:59:40,569 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-4) [31c37a79] Correlation ID: 31c37a79, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-09-10 12:59:40,585 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-3) [63537fc9] Correlation ID: 63537fc9, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-09-10 12:59:40,634 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp-/127.0.0.1:8702-9) [55c76991] Running command: LogoutUserCommand internal: false. >2014-09-10 12:59:40,652 INFO [org.ovirt.engine.ui.frontend.server.gwt.plugin.PluginDataManager] (ajp-/127.0.0.1:8702-1) Reading UI plugin descriptor [/usr/share/ovirt-engine/ui-plugins/redhat_support_plugin_rhev.json] >2014-09-10 12:59:40,665 INFO [org.ovirt.engine.ui.frontend.server.gwt.plugin.PluginDataManager] (ajp-/127.0.0.1:8702-1) Reading UI plugin configuration [/etc/ovirt-engine/ui-plugins/redhat_support_plugin_rhev-config.json] >2014-09-10 12:59:40,670 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp-/127.0.0.1:8702-10) [132af7bc] Running command: LogoutUserCommand internal: false. >2014-09-10 12:59:40,711 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-9) [55c76991] Correlation ID: 55c76991, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-09-10 12:59:40,731 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-10) [132af7bc] Correlation ID: 132af7bc, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-09-10 12:59:41,003 INFO [org.ovirt.engine.core.utils.servlet.UnsupportedLocaleHelper] (ajp-/127.0.0.1:8702-1) Invalid locale found in configuration: >2014-09-10 12:59:41,003 INFO [org.ovirt.engine.core.utils.servlet.UnsupportedLocaleHelper] (ajp-/127.0.0.1:8702-1) Invalid locale found in configuration: >2014-09-10 12:59:55,264 INFO [org.ovirt.engine.core.bll.aaa.LoginAdminUserCommand] (ajp-/127.0.0.1:8702-8) Running command: LoginAdminUserCommand internal: false. >2014-09-10 12:59:55,281 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-8) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-09-10 13:00:00,125 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-93) Autorecovering 1 hosts >2014-09-10 13:00:00,125 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-93) Autorecovering hosts id: 7ebdb254-efb6-480b-8c4e-a7dbdc759ed4, name : rack >2014-09-10 13:00:00,135 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-93) [92dad18] Lock Acquired to object EngineLock [exclusiveLocks= key: 7ebdb254-efb6-480b-8c4e-a7dbdc759ed4 value: VDS >, sharedLocks= ] >2014-09-10 13:00:00,136 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-93) [92dad18] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 7ebdb254-efb6-480b-8c4e-a7dbdc759ed4 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2014-09-10 13:00:00,136 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-93) [92dad18] Before acquiring lock in order to prevent monitoring for host rack from data-center Default >2014-09-10 13:00:00,136 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-93) [92dad18] Lock acquired, from now a monitoring of host will be skipped for host rack from data-center Default >2014-09-10 13:00:00,284 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-93) [92dad18] START, SetVdsStatusVDSCommand(HostName = rack, HostId = 7ebdb254-efb6-480b-8c4e-a7dbdc759ed4, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 12dbc0c2 >2014-09-10 13:00:00,328 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-93) [92dad18] FINISH, SetVdsStatusVDSCommand, log id: 12dbc0c2 >2014-09-10 13:00:00,561 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-93) [92dad18] Activate finished. Lock released. Monitoring can run now for host rack from data-center Default >2014-09-10 13:00:00,606 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-93) [92dad18] Correlation ID: 92dad18, Call Stack: null, Custom Event ID: -1, Message: Host rack was autorecovered. >2014-09-10 13:00:00,607 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-93) [92dad18] Lock freed to object EngineLock [exclusiveLocks= key: 7ebdb254-efb6-480b-8c4e-a7dbdc759ed4 value: VDS >, sharedLocks= ] >2014-09-10 13:00:00,786 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-96) START, GetHardwareInfoVDSCommand(HostName = rack, HostId = 7ebdb254-efb6-480b-8c4e-a7dbdc759ed4, vds=Host[rack,7ebdb254-efb6-480b-8c4e-a7dbdc759ed4]), log id: 65ae2ff0 >2014-09-10 13:00:00,792 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-96) FINISH, GetHardwareInfoVDSCommand, log id: 65ae2ff0 >2014-09-10 13:00:01,105 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-96) [67050309] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 7ebdb254-efb6-480b-8c4e-a7dbdc759ed4 Type: VDS >2014-09-10 13:00:01,110 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-96) [67050309] START, SetVdsStatusVDSCommand(HostName = rack, HostId = 7ebdb254-efb6-480b-8c4e-a7dbdc759ed4, status=NonOperational, nonOperationalReason=NETWORK_UNREACHABLE, stopSpmFailureLogged=false), log id: 4ac4c6c7 >2014-09-10 13:00:01,155 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-96) [67050309] FINISH, SetVdsStatusVDSCommand, log id: 4ac4c6c7 >2014-09-10 13:00:01,162 ERROR [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-96) [67050309] Host rack is set to Non-Operational, it is missing the following networks: rhevm >2014-09-10 13:00:01,223 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-96) [3b8ca6e0] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 7ebdb254-efb6-480b-8c4e-a7dbdc759ed4 Type: VDS >2014-09-10 13:00:01,248 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-96) [3b8ca6e0] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host rack was set to NonOperational. >2014-09-10 13:00:01,354 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-96) [21e034cd] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 7ebdb254-efb6-480b-8c4e-a7dbdc759ed4 Type: VDS >2014-09-10 13:00:01,360 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-96) [21e034cd] Host 7ebdb254-efb6-480b-8c4e-a7dbdc759ed4 : rack is already in NonOperational status for reason NETWORK_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-09-10 13:00:24,594 INFO [org.ovirt.engine.core.bll.MaintenanceNumberOfVdssCommand] (org.ovirt.thread.pool-6-thread-9) [6e736f53] Running command: MaintenanceNumberOfVdssCommand internal: false. Entities affected : ID: 7ebdb254-efb6-480b-8c4e-a7dbdc759ed4 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2014-09-10 13:00:24,612 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-6-thread-9) [6e736f53] START, SetVdsStatusVDSCommand(HostName = rack, HostId = 7ebdb254-efb6-480b-8c4e-a7dbdc759ed4, status=PreparingForMaintenance, nonOperationalReason=NONE, stopSpmFailureLogged=true), log id: 53d16b1c >2014-09-10 13:00:24,663 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-6-thread-9) [6e736f53] FINISH, SetVdsStatusVDSCommand, log id: 53d16b1c >2014-09-10 13:00:24,958 INFO [org.ovirt.engine.core.bll.MaintenanceVdsCommand] (org.ovirt.thread.pool-6-thread-9) [6e736f53] Running command: MaintenanceVdsCommand internal: true. Entities affected : ID: 7ebdb254-efb6-480b-8c4e-a7dbdc759ed4 Type: VDS >2014-09-10 13:00:24,974 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-9) [6e736f53] Correlation ID: 6e736f53, Job ID: ed1d360e-de71-4da3-84f6-bece0777a159, Call Stack: null, Custom Event ID: -1, Message: Host rack was switched to Maintenance mode by admin. >2014-09-10 13:00:25,529 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-56) Updated vds status from Preparing for Maintenance to Maintenance in database, vds = 7ebdb254-efb6-480b-8c4e-a7dbdc759ed4 : rack >2014-09-10 13:00:25,660 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-6-thread-11) Clearing cache of pool: 00000002-0002-0002-0002-0000000001ad for problematic entities of VDS: rack. >2014-09-10 13:00:25,660 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-6-thread-11) Removing vds [7ebdb254-efb6-480b-8c4e-a7dbdc759ed4] from the domain in maintenance cache >2014-09-10 13:00:25,660 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-6-thread-11) Removing host(s) [7ebdb254-efb6-480b-8c4e-a7dbdc759ed4] from hosts unseen domain report cache >2014-09-10 13:00:36,175 INFO [org.ovirt.engine.core.bll.RemoveVdsCommand] (ajp-/127.0.0.1:8702-3) [60fa1c6d] Lock Acquired to object EngineLock [exclusiveLocks= key: 7ebdb254-efb6-480b-8c4e-a7dbdc759ed4 value: VDS >, sharedLocks= ] >2014-09-10 13:00:36,229 INFO [org.ovirt.engine.core.bll.RemoveVdsCommand] (org.ovirt.thread.pool-6-thread-13) [60fa1c6d] Running command: RemoveVdsCommand internal: false. Entities affected : ID: 7ebdb254-efb6-480b-8c4e-a7dbdc759ed4 Type: VDSAction group DELETE_HOST with role type ADMIN >2014-09-10 13:00:36,357 INFO [org.ovirt.engine.core.vdsbroker.RemoveVdsVDSCommand] (org.ovirt.thread.pool-6-thread-13) [60fa1c6d] START, RemoveVdsVDSCommand( HostId = 7ebdb254-efb6-480b-8c4e-a7dbdc759ed4), log id: 69101be1 >2014-09-10 13:00:36,357 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (org.ovirt.thread.pool-6-thread-13) [60fa1c6d] vdsManager::disposing >2014-09-10 13:00:36,362 INFO [org.ovirt.engine.core.vdsbroker.RemoveVdsVDSCommand] (org.ovirt.thread.pool-6-thread-13) [60fa1c6d] FINISH, RemoveVdsVDSCommand, log id: 69101be1 >2014-09-10 13:00:36,373 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-13) [60fa1c6d] Correlation ID: 60fa1c6d, Call Stack: null, Custom Event ID: -1, Message: Host rack was removed by admin. >2014-09-10 13:00:36,373 INFO [org.ovirt.engine.core.bll.RemoveVdsCommand] (org.ovirt.thread.pool-6-thread-13) [60fa1c6d] Lock freed to object EngineLock [exclusiveLocks= key: 7ebdb254-efb6-480b-8c4e-a7dbdc759ed4 value: VDS >, sharedLocks= ] >2014-09-10 13:00:58,564 INFO [org.ovirt.engine.core.bll.AddVdsCommand] (ajp-/127.0.0.1:8702-4) [523871a6] Running command: AddVdsCommand internal: false. Entities affected : ID: 00000001-0001-0001-0001-000000000399 Type: VdsGroupsAction group CREATE_HOST with role type ADMIN >2014-09-10 13:00:58,753 INFO [org.ovirt.engine.core.bll.AddVdsSpmIdCommand] (ajp-/127.0.0.1:8702-4) [349d74fb] Running command: AddVdsSpmIdCommand internal: true. Entities affected : ID: 62d62bcc-dea3-4792-b313-9dcd4da8d3dc Type: VDS >2014-09-10 13:00:58,818 INFO [org.ovirt.engine.core.bll.AddVdsSpmIdCommand] (ajp-/127.0.0.1:8702-4) [349d74fb] Lock freed to object EngineLock [exclusiveLocks= key: 00000002-0002-0002-0002-0000000001ad value: REGISTER_VDS >, sharedLocks= ] >2014-09-10 13:00:58,822 INFO [org.ovirt.engine.core.vdsbroker.RemoveVdsVDSCommand] (ajp-/127.0.0.1:8702-4) [349d74fb] START, RemoveVdsVDSCommand(HostName = rack, HostId = 62d62bcc-dea3-4792-b313-9dcd4da8d3dc), log id: 1aa5cd79 >2014-09-10 13:00:58,824 INFO [org.ovirt.engine.core.vdsbroker.RemoveVdsVDSCommand] (ajp-/127.0.0.1:8702-4) [349d74fb] FINISH, RemoveVdsVDSCommand, log id: 1aa5cd79 >2014-09-10 13:00:58,828 INFO [org.ovirt.engine.core.vdsbroker.AddVdsVDSCommand] (ajp-/127.0.0.1:8702-4) [349d74fb] START, AddVdsVDSCommand(HostName = rack, HostId = 62d62bcc-dea3-4792-b313-9dcd4da8d3dc), log id: 58229403 >2014-09-10 13:00:58,829 INFO [org.ovirt.engine.core.vdsbroker.AddVdsVDSCommand] (ajp-/127.0.0.1:8702-4) [349d74fb] AddVds - entered , starting logic to add VDS 62d62bcc-dea3-4792-b313-9dcd4da8d3dc >2014-09-10 13:00:58,831 INFO [org.ovirt.engine.core.vdsbroker.AddVdsVDSCommand] (ajp-/127.0.0.1:8702-4) [349d74fb] AddVds - VDS 62d62bcc-dea3-4792-b313-9dcd4da8d3dc was added, will try to add it to the resource manager >2014-09-10 13:00:58,831 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (ajp-/127.0.0.1:8702-4) [349d74fb] Entered VdsManager constructor >2014-09-10 13:00:58,848 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (ajp-/127.0.0.1:8702-4) [349d74fb] Initialize vdsBroker (dell-r210ii-16.rhev.lab.eng.brq.redhat.com,54,321) >2014-09-10 13:00:59,018 INFO [org.ovirt.engine.core.vdsbroker.ResourceManager] (ajp-/127.0.0.1:8702-4) [349d74fb] VDS 62d62bcc-dea3-4792-b313-9dcd4da8d3dc was added to the Resource Manager >2014-09-10 13:00:59,019 INFO [org.ovirt.engine.core.vdsbroker.AddVdsVDSCommand] (ajp-/127.0.0.1:8702-4) [349d74fb] FINISH, AddVdsVDSCommand, log id: 58229403 >2014-09-10 13:00:59,039 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-4) [349d74fb] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host rack. >2014-09-10 13:00:59,110 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-4) [349d74fb] Correlation ID: 523871a6, Job ID: bdfdc66c-d14e-4d7a-9e9f-fe7e50b4f8d0, Call Stack: null, Custom Event ID: -1, Message: Host rack was added by admin. >2014-09-10 13:00:59,115 INFO [org.ovirt.engine.core.bll.InstallVdsInternalCommand] (org.ovirt.thread.pool-6-thread-14) [6fbdab31] Lock Acquired to object EngineLock [exclusiveLocks= key: 62d62bcc-dea3-4792-b313-9dcd4da8d3dc value: VDS >, sharedLocks= ] >2014-09-10 13:00:59,139 INFO [org.ovirt.engine.core.bll.InstallVdsInternalCommand] (org.ovirt.thread.pool-6-thread-14) [6fbdab31] Running command: InstallVdsInternalCommand internal: true. Entities affected : ID: 62d62bcc-dea3-4792-b313-9dcd4da8d3dc Type: VDS >2014-09-10 13:00:59,148 INFO [org.ovirt.engine.core.bll.InstallVdsInternalCommand] (org.ovirt.thread.pool-6-thread-14) [6fbdab31] Before Installation host 62d62bcc-dea3-4792-b313-9dcd4da8d3dc, rack >2014-09-10 13:00:59,159 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-6-thread-14) [6fbdab31] START, SetVdsStatusVDSCommand(HostName = rack, HostId = 62d62bcc-dea3-4792-b313-9dcd4da8d3dc, status=Installing, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 6cf90d7d >2014-09-10 13:00:59,170 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-6-thread-14) [6fbdab31] FINISH, SetVdsStatusVDSCommand, log id: 6cf90d7d >2014-09-10 13:00:59,220 INFO [org.ovirt.engine.core.bll.InstallerMessages] (org.ovirt.thread.pool-6-thread-14) [6fbdab31] Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Connected to host dell-r210ii-16.rhev.lab.eng.brq.redhat.com with SSH key fingerprint: 25:12:7b:de:30:1d:b3:5e:bf:d2:a6:c7:37:77:d3:b1 >2014-09-10 13:00:59,247 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-14) [6fbdab31] Correlation ID: 6fbdab31, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Connected to host dell-r210ii-16.rhev.lab.eng.brq.redhat.com with SSH key fingerprint: 25:12:7b:de:30:1d:b3:5e:bf:d2:a6:c7:37:77:d3:b1. >2014-09-10 13:00:59,266 INFO [org.ovirt.engine.core.bll.VdsDeploy] (org.ovirt.thread.pool-6-thread-14) [6fbdab31] Installation of dell-r210ii-16.rhev.lab.eng.brq.redhat.com. Executing command via SSH umask 0077; MYTMP="$(TMPDIR="${OVIRT_TMPDIR}" mktemp -t ovirt-XXXXXXXXXX)"; trap "chmod -R u+rwX \"${MYTMP}\" > /dev/null 2>&1; rm -fr \"${MYTMP}\" > /dev/null 2>&1" 0; rm -fr "${MYTMP}" && mkdir "${MYTMP}" && tar --warning=no-timestamp -C "${MYTMP}" -x && "${MYTMP}"/setup DIALOG/dialect=str:machine DIALOG/customization=bool:True < /var/cache/ovirt-engine/ovirt-host-deploy.tar >2014-09-10 13:00:59,278 INFO [org.ovirt.engine.core.uutils.ssh.SSHDialog] (org.ovirt.thread.pool-6-thread-14) SSH execute root@dell-r210ii-16.rhev.lab.eng.brq.redhat.com 'umask 0077; MYTMP="$(TMPDIR="${OVIRT_TMPDIR}" mktemp -t ovirt-XXXXXXXXXX)"; trap "chmod -R u+rwX \"${MYTMP}\" > /dev/null 2>&1; rm -fr \"${MYTMP}\" > /dev/null 2>&1" 0; rm -fr "${MYTMP}" && mkdir "${MYTMP}" && tar --warning=no-timestamp -C "${MYTMP}" -x && "${MYTMP}"/setup DIALOG/dialect=str:machine DIALOG/customization=bool:True' >2014-09-10 13:01:00,391 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Stage: Initializing >2014-09-10 13:01:00,407 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6fbdab31, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Stage: Initializing. >2014-09-10 13:01:00,419 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Stage: Environment setup >2014-09-10 13:01:00,433 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6fbdab31, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Stage: Environment setup. >2014-09-10 13:01:00,434 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Stage: Environment packages setup >2014-09-10 13:01:00,446 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6fbdab31, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Stage: Environment packages setup. >2014-09-10 13:01:02,023 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-26) Initializing Host: rack >2014-09-10 13:01:11,375 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Stage: Programs detection >2014-09-10 13:01:11,389 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6fbdab31, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Stage: Programs detection. >2014-09-10 13:01:11,390 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Stage: Environment customization >2014-09-10 13:01:11,406 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6fbdab31, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Stage: Environment customization. >2014-09-10 13:01:11,773 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Kdump supported >2014-09-10 13:01:11,787 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6fbdab31, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Kdump supported. >2014-09-10 13:01:11,806 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Logs at host located at: '/tmp/ovirt-host-deploy-20140910130059-bknidl.log' >2014-09-10 13:01:11,828 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6fbdab31, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Logs at host located at: '/tmp/ovirt-host-deploy-20140910130059-bknidl.log'. >2014-09-10 13:01:11,880 INFO [org.ovirt.engine.core.bll.VdsDeploy] (VdsDeploy) Host dell-r210ii-16.rhev.lab.eng.brq.redhat.com reports unique id 4C4C4544-0034-5310-8052-B4C04F4A354A >2014-09-10 13:01:11,911 INFO [org.ovirt.engine.core.bll.VdsDeploy] (VdsDeploy) Assigning unique id 4C4C4544-0034-5310-8052-B4C04F4A354A to Host dell-r210ii-16.rhev.lab.eng.brq.redhat.com >2014-09-10 13:01:12,039 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Disabling Kdump integration >2014-09-10 13:01:12,059 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6fbdab31, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Disabling Kdump integration. >2014-09-10 13:01:12,086 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Stage: Setup validation >2014-09-10 13:01:12,107 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6fbdab31, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Stage: Setup validation. >2014-09-10 13:01:12,125 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Hardware supports virtualization >2014-09-10 13:01:12,144 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6fbdab31, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Hardware supports virtualization. >2014-09-10 13:01:12,370 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Stage: Transaction setup >2014-09-10 13:01:12,384 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6fbdab31, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Stage: Transaction setup. >2014-09-10 13:01:12,423 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Stage: Misc configuration >2014-09-10 13:01:12,433 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6fbdab31, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Stage: Misc configuration. >2014-09-10 13:01:12,433 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Stage: Package installation >2014-09-10 13:01:12,441 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6fbdab31, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Stage: Package installation. >2014-09-10 13:01:19,691 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Stage: Misc configuration >2014-09-10 13:01:19,707 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6fbdab31, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Stage: Misc configuration. >2014-09-10 13:01:19,707 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Setting time >2014-09-10 13:01:19,720 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6fbdab31, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Setting time. >2014-09-10 13:01:21,198 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Enrolling certificate >2014-09-10 13:01:21,211 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6fbdab31, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Enrolling certificate. >2014-09-10 13:01:22,457 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Stage: Transaction commit >2014-09-10 13:01:22,486 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6fbdab31, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Stage: Transaction commit. >2014-09-10 13:01:22,526 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Stage: Closing up >2014-09-10 13:01:22,540 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6fbdab31, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Stage: Closing up. >2014-09-10 13:01:23,293 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Stopping libvirtd >2014-09-10 13:01:23,310 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6fbdab31, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Stopping libvirtd. >2014-09-10 13:01:23,567 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Starting vdsm >2014-09-10 13:01:23,581 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6fbdab31, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Starting vdsm. >2014-09-10 13:01:25,368 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Stage: Pre-termination >2014-09-10 13:01:25,383 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6fbdab31, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Stage: Pre-termination. >2014-09-10 13:01:25,398 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Retrieving installation logs to: '/var/log/ovirt-engine/host-deploy/ovirt-20140910130125-dell-r210ii-16.rhev.lab.eng.brq.redhat.com-6fbdab31.log' >2014-09-10 13:01:25,416 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6fbdab31, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Retrieving installation logs to: '/var/log/ovirt-engine/host-deploy/ovirt-20140910130125-dell-r210ii-16.rhev.lab.eng.brq.redhat.com-6fbdab31.log'. >2014-09-10 13:01:25,664 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Stage: Termination >2014-09-10 13:01:25,682 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 6fbdab31, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Stage: Termination. >2014-09-10 13:01:26,010 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.PollVDSCommand] (org.ovirt.thread.pool-6-thread-14) [6fbdab31] org.ovirt.engine.core.vdsbroker.vdsbroker.VDSRecoveringException: Recovering from crash or Initializing >2014-09-10 13:01:26,042 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.PollVDSCommand] (org.ovirt.thread.pool-6-thread-14) [6fbdab31] Command PollVDSCommand(HostName = rack, HostId = 62d62bcc-dea3-4792-b313-9dcd4da8d3dc) execution failed. Exception: VDSRecoveringException: Recovering from crash or Initializing >2014-09-10 13:01:26,588 INFO [org.ovirt.engine.core.bll.network.NetworkConfigurator] (org.ovirt.thread.pool-6-thread-14) [6fbdab31] Engine managed to communicate with VDSM agent on host rack >2014-09-10 13:01:27,331 INFO [org.ovirt.engine.core.bll.network.host.SetupNetworksCommand] (org.ovirt.thread.pool-6-thread-14) [2ef40273] Running command: SetupNetworksCommand internal: true. Entities affected : ID: 62d62bcc-dea3-4792-b313-9dcd4da8d3dc Type: VDSAction group CONFIGURE_HOST_NETWORK with role type ADMIN >2014-09-10 13:01:27,338 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetupNetworksVDSCommand] (org.ovirt.thread.pool-6-thread-14) [2ef40273] START, SetupNetworksVDSCommand(HostName = rack, HostId = 62d62bcc-dea3-4792-b313-9dcd4da8d3dc, force=false, checkConnectivity=true, conectivityTimeout=120, > networks=[rhevm {id=00000000-0000-0000-0000-000000000009, description=Management Network, comment=null, subnet=null, gateway=null, type=null, vlanId=null, stp=false, dataCenterId=00000002-0002-0002-0002-0000000001ad, mtu=0, vmNetwork=true, cluster=NetworkCluster {id={clusterId=null, networkId=null}, status=OPERATIONAL, display=true, required=true, migration=true}, providedBy=null, label=null, qosId=null}], > bonds=[], > interfaces=[em1 {id=2ae645fb-b95f-4ac7-bf44-d6ea67f22157, vdsId=62d62bcc-dea3-4792-b313-9dcd4da8d3dc, macAddress=d4:ae:52:c7:0d:8b, networkName=rhevm, baseInterface=null, vlanId=null, bonded=null, bondName=null, bondOptions=null, bootProtocol=DHCP, address=10.34.62.208, subnet=255.255.252.0, gateway=null, mtu=1500, bridged=false, speed=1000, type=0, networkImplementationDetails=null, qos=null qosOverridden=false, labels=null, customProperties=null}, > em2 {id=bab47b56-1de3-4e38-b789-a162c716830b, vdsId=62d62bcc-dea3-4792-b313-9dcd4da8d3dc, macAddress=d4:ae:52:c7:0d:8c, networkName=null, baseInterface=null, vlanId=null, bonded=null, bondName=null, bondOptions=null, bootProtocol=DHCP, address=, subnet=, gateway=null, mtu=1500, bridged=false, speed=0, type=0, networkImplementationDetails=null, qos=null qosOverridden=false, labels=null, customProperties=null}], > removedNetworks=[], > removedBonds=[]), log id: 16957cae >2014-09-10 13:01:27,343 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetupNetworksVDSCommand] (org.ovirt.thread.pool-6-thread-14) [2ef40273] FINISH, SetupNetworksVDSCommand, log id: 16957cae >2014-09-10 13:01:30,924 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.PollVDSCommand] (org.ovirt.thread.pool-6-thread-14) [2ef40273] Command PollVDSCommand(HostName = rack, HostId = 62d62bcc-dea3-4792-b313-9dcd4da8d3dc) execution failed. Exception: VDSNetworkException: VDSGenericException: VDSNetworkException: Timeout during xml-rpc call >2014-09-10 13:01:30,924 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.PollVDSCommand] (org.ovirt.thread.pool-6-thread-14) [2ef40273] Timeout waiting for VDSM response. java.util.concurrent.TimeoutException >2014-09-10 13:01:33,932 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.PollVDSCommand] (org.ovirt.thread.pool-6-thread-14) [2ef40273] Command PollVDSCommand(HostName = rack, HostId = 62d62bcc-dea3-4792-b313-9dcd4da8d3dc) execution failed. Exception: VDSNetworkException: VDSGenericException: VDSNetworkException: Timeout during xml-rpc call >2014-09-10 13:01:33,932 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.PollVDSCommand] (org.ovirt.thread.pool-6-thread-14) [2ef40273] Timeout waiting for VDSM response. java.util.concurrent.TimeoutException >2014-09-10 13:01:36,936 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.PollVDSCommand] (org.ovirt.thread.pool-6-thread-14) [2ef40273] Command PollVDSCommand(HostName = rack, HostId = 62d62bcc-dea3-4792-b313-9dcd4da8d3dc) execution failed. Exception: VDSNetworkException: VDSGenericException: VDSNetworkException: Timeout during xml-rpc call >2014-09-10 13:01:36,937 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.PollVDSCommand] (org.ovirt.thread.pool-6-thread-14) [2ef40273] Timeout waiting for VDSM response. java.util.concurrent.TimeoutException >2014-09-10 13:01:39,946 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.PollVDSCommand] (org.ovirt.thread.pool-6-thread-14) [2ef40273] Command PollVDSCommand(HostName = rack, HostId = 62d62bcc-dea3-4792-b313-9dcd4da8d3dc) execution failed. Exception: VDSNetworkException: VDSGenericException: VDSNetworkException: Timeout during xml-rpc call >2014-09-10 13:01:39,946 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.PollVDSCommand] (org.ovirt.thread.pool-6-thread-14) [2ef40273] Timeout waiting for VDSM response. java.util.concurrent.TimeoutException >2014-09-10 13:01:41,976 INFO [org.ovirt.engine.core.bll.network.host.CommitNetworkChangesCommand] (org.ovirt.thread.pool-6-thread-14) [72bc084] Running command: CommitNetworkChangesCommand internal: true. Entities affected : ID: 62d62bcc-dea3-4792-b313-9dcd4da8d3dc Type: VDSAction group CONFIGURE_HOST_NETWORK with role type ADMIN >2014-09-10 13:01:41,979 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetSafeNetworkConfigVDSCommand] (org.ovirt.thread.pool-6-thread-14) [72bc084] START, SetSafeNetworkConfigVDSCommand(HostName = rack, HostId = 62d62bcc-dea3-4792-b313-9dcd4da8d3dc), log id: 1cff268c >2014-09-10 13:01:42,032 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetSafeNetworkConfigVDSCommand] (org.ovirt.thread.pool-6-thread-14) [72bc084] FINISH, SetSafeNetworkConfigVDSCommand, log id: 1cff268c >2014-09-10 13:01:42,055 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-14) [72bc084] Correlation ID: 72bc084, Call Stack: null, Custom Event ID: -1, Message: Network changes were saved on host rack >2014-09-10 13:01:42,058 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-6-thread-14) [72bc084] START, SetVdsStatusVDSCommand(HostName = rack, HostId = 62d62bcc-dea3-4792-b313-9dcd4da8d3dc, status=Initializing, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 455dae10 >2014-09-10 13:01:42,088 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-6-thread-14) [72bc084] FINISH, SetVdsStatusVDSCommand, log id: 455dae10 >2014-09-10 13:01:42,088 INFO [org.ovirt.engine.core.bll.InstallVdsInternalCommand] (org.ovirt.thread.pool-6-thread-14) [72bc084] After Installation host rack, VDS >2014-09-10 13:01:42,112 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-14) [72bc084] Correlation ID: 6fbdab31, Job ID: bdfdc66c-d14e-4d7a-9e9f-fe7e50b4f8d0, Call Stack: null, Custom Event ID: -1, Message: Host rack installed >2014-09-10 13:01:42,123 INFO [org.ovirt.engine.core.bll.InstallVdsInternalCommand] (org.ovirt.thread.pool-6-thread-14) [72bc084] Lock freed to object EngineLock [exclusiveLocks= key: 62d62bcc-dea3-4792-b313-9dcd4da8d3dc value: VDS >, sharedLocks= ] >2014-09-10 13:01:44,413 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-20) START, GetHardwareInfoVDSCommand(HostName = rack, HostId = 62d62bcc-dea3-4792-b313-9dcd4da8d3dc, vds=Host[rack,62d62bcc-dea3-4792-b313-9dcd4da8d3dc]), log id: 6fc4d38e >2014-09-10 13:01:44,492 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-20) FINISH, GetHardwareInfoVDSCommand, log id: 6fc4d38e >2014-09-10 13:01:44,540 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-20) [53caa8d2] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 62d62bcc-dea3-4792-b313-9dcd4da8d3dc Type: VDS >2014-09-10 13:01:44,889 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-20) [53caa8d2] START, GetHardwareInfoVDSCommand(HostName = rack, HostId = 62d62bcc-dea3-4792-b313-9dcd4da8d3dc, vds=Host[rack,62d62bcc-dea3-4792-b313-9dcd4da8d3dc]), log id: 3b551c82 >2014-09-10 13:01:44,947 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-20) [53caa8d2] FINISH, GetHardwareInfoVDSCommand, log id: 3b551c82 >2014-09-10 13:01:45,093 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-20) [6c9eb5a6] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001ad Type: StoragePool >2014-09-10 13:01:45,139 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-20) [6c9eb5a6] START, SetMOMPolicyParametersVDSCommand(HostName = rack, HostId = 62d62bcc-dea3-4792-b313-9dcd4da8d3dc), log id: 76b4dea0 >2014-09-10 13:01:45,278 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-20) [6c9eb5a6] FINISH, SetMOMPolicyParametersVDSCommand, log id: 76b4dea0 >2014-09-10 13:01:45,304 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-20) [6c9eb5a6] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host rack was set to Up. >2014-09-10 13:01:45,324 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-20) [6c9eb5a6] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host rack. >2014-09-10 13:01:45,391 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-20) [30a2fdfe] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 62d62bcc-dea3-4792-b313-9dcd4da8d3dc Type: VDS >2014-09-10 13:01:45,413 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-20) [232c3800] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 62d62bcc-dea3-4792-b313-9dcd4da8d3dc Type: VDS >2014-09-10 13:22:55,293 INFO [org.ovirt.engine.core.bll.MaintenanceNumberOfVdssCommand] (org.ovirt.thread.pool-6-thread-41) [7469b73c] Running command: MaintenanceNumberOfVdssCommand internal: false. Entities affected : ID: 62d62bcc-dea3-4792-b313-9dcd4da8d3dc Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2014-09-10 13:22:55,303 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-6-thread-41) [7469b73c] START, SetVdsStatusVDSCommand(HostName = rack, HostId = 62d62bcc-dea3-4792-b313-9dcd4da8d3dc, status=PreparingForMaintenance, nonOperationalReason=NONE, stopSpmFailureLogged=true), log id: 50b027ab >2014-09-10 13:22:55,333 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-6-thread-41) [7469b73c] FINISH, SetVdsStatusVDSCommand, log id: 50b027ab >2014-09-10 13:22:55,434 INFO [org.ovirt.engine.core.bll.MaintenanceVdsCommand] (org.ovirt.thread.pool-6-thread-41) [7469b73c] Running command: MaintenanceVdsCommand internal: true. Entities affected : ID: 62d62bcc-dea3-4792-b313-9dcd4da8d3dc Type: VDS >2014-09-10 13:22:55,451 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-41) [7469b73c] Correlation ID: 7469b73c, Job ID: a6a2da1a-d1a5-47b9-82e6-bebfa3cb288f, Call Stack: null, Custom Event ID: -1, Message: Host rack was switched to Maintenance mode by admin. >2014-09-10 13:22:58,393 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-92) Updated vds status from Preparing for Maintenance to Maintenance in database, vds = 62d62bcc-dea3-4792-b313-9dcd4da8d3dc : rack >2014-09-10 13:22:58,436 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-6-thread-45) Clearing cache of pool: 00000002-0002-0002-0002-0000000001ad for problematic entities of VDS: rack. >2014-09-10 13:22:58,436 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-6-thread-45) Removing vds [62d62bcc-dea3-4792-b313-9dcd4da8d3dc] from the domain in maintenance cache >2014-09-10 13:22:58,436 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-6-thread-45) Removing host(s) [62d62bcc-dea3-4792-b313-9dcd4da8d3dc] from hosts unseen domain report cache >2014-09-10 13:23:18,768 INFO [org.ovirt.engine.core.bll.RemoveVdsCommand] (ajp-/127.0.0.1:8702-10) [5ce283d7] Lock Acquired to object EngineLock [exclusiveLocks= key: 62d62bcc-dea3-4792-b313-9dcd4da8d3dc value: VDS >, sharedLocks= ] >2014-09-10 13:23:18,780 INFO [org.ovirt.engine.core.bll.RemoveVdsCommand] (org.ovirt.thread.pool-6-thread-47) [5ce283d7] Running command: RemoveVdsCommand internal: false. Entities affected : ID: 62d62bcc-dea3-4792-b313-9dcd4da8d3dc Type: VDSAction group DELETE_HOST with role type ADMIN >2014-09-10 13:23:18,836 INFO [org.ovirt.engine.core.vdsbroker.RemoveVdsVDSCommand] (org.ovirt.thread.pool-6-thread-47) [5ce283d7] START, RemoveVdsVDSCommand( HostId = 62d62bcc-dea3-4792-b313-9dcd4da8d3dc), log id: 5633cab9 >2014-09-10 13:23:18,836 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (org.ovirt.thread.pool-6-thread-47) [5ce283d7] vdsManager::disposing >2014-09-10 13:23:18,838 INFO [org.ovirt.engine.core.vdsbroker.RemoveVdsVDSCommand] (org.ovirt.thread.pool-6-thread-47) [5ce283d7] FINISH, RemoveVdsVDSCommand, log id: 5633cab9 >2014-09-10 13:23:18,847 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-47) [5ce283d7] Correlation ID: 5ce283d7, Call Stack: null, Custom Event ID: -1, Message: Host rack was removed by admin. >2014-09-10 13:23:18,847 INFO [org.ovirt.engine.core.bll.RemoveVdsCommand] (org.ovirt.thread.pool-6-thread-47) [5ce283d7] Lock freed to object EngineLock [exclusiveLocks= key: 62d62bcc-dea3-4792-b313-9dcd4da8d3dc value: VDS >, sharedLocks= ] >2014-09-10 13:23:51,765 INFO [org.ovirt.engine.core.bll.AddVdsCommand] (ajp-/127.0.0.1:8702-1) [68a704d0] Running command: AddVdsCommand internal: false. Entities affected : ID: 00000001-0001-0001-0001-000000000399 Type: VdsGroupsAction group CREATE_HOST with role type ADMIN >2014-09-10 13:23:51,825 INFO [org.ovirt.engine.core.bll.AddVdsSpmIdCommand] (ajp-/127.0.0.1:8702-1) [41a5b178] Running command: AddVdsSpmIdCommand internal: true. Entities affected : ID: f689c28a-07de-4408-ac86-81382b6dd288 Type: VDS >2014-09-10 13:23:51,843 INFO [org.ovirt.engine.core.bll.AddVdsSpmIdCommand] (ajp-/127.0.0.1:8702-1) [41a5b178] Lock freed to object EngineLock [exclusiveLocks= key: 00000002-0002-0002-0002-0000000001ad value: REGISTER_VDS >, sharedLocks= ] >2014-09-10 13:23:51,847 INFO [org.ovirt.engine.core.vdsbroker.RemoveVdsVDSCommand] (ajp-/127.0.0.1:8702-1) [41a5b178] START, RemoveVdsVDSCommand(HostName = rack, HostId = f689c28a-07de-4408-ac86-81382b6dd288), log id: 75f2fae4 >2014-09-10 13:23:51,847 INFO [org.ovirt.engine.core.vdsbroker.RemoveVdsVDSCommand] (ajp-/127.0.0.1:8702-1) [41a5b178] FINISH, RemoveVdsVDSCommand, log id: 75f2fae4 >2014-09-10 13:23:51,849 INFO [org.ovirt.engine.core.vdsbroker.AddVdsVDSCommand] (ajp-/127.0.0.1:8702-1) [41a5b178] START, AddVdsVDSCommand(HostName = rack, HostId = f689c28a-07de-4408-ac86-81382b6dd288), log id: 7eecc7e3 >2014-09-10 13:23:51,849 INFO [org.ovirt.engine.core.vdsbroker.AddVdsVDSCommand] (ajp-/127.0.0.1:8702-1) [41a5b178] AddVds - entered , starting logic to add VDS f689c28a-07de-4408-ac86-81382b6dd288 >2014-09-10 13:23:51,851 INFO [org.ovirt.engine.core.vdsbroker.AddVdsVDSCommand] (ajp-/127.0.0.1:8702-1) [41a5b178] AddVds - VDS f689c28a-07de-4408-ac86-81382b6dd288 was added, will try to add it to the resource manager >2014-09-10 13:23:51,851 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (ajp-/127.0.0.1:8702-1) [41a5b178] Entered VdsManager constructor >2014-09-10 13:23:51,865 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (ajp-/127.0.0.1:8702-1) [41a5b178] Initialize vdsBroker (dell-r210ii-16.rhev.lab.eng.brq.redhat.com,54,321) >2014-09-10 13:23:51,866 INFO [org.ovirt.engine.core.vdsbroker.ResourceManager] (ajp-/127.0.0.1:8702-1) [41a5b178] VDS f689c28a-07de-4408-ac86-81382b6dd288 was added to the Resource Manager >2014-09-10 13:23:51,866 INFO [org.ovirt.engine.core.vdsbroker.AddVdsVDSCommand] (ajp-/127.0.0.1:8702-1) [41a5b178] FINISH, AddVdsVDSCommand, log id: 7eecc7e3 >2014-09-10 13:23:51,879 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-1) [41a5b178] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host rack. >2014-09-10 13:23:51,912 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-1) [41a5b178] Correlation ID: 68a704d0, Job ID: f04c0777-cd59-4cea-8771-d9946f27e6e2, Call Stack: null, Custom Event ID: -1, Message: Host rack was added by admin. >2014-09-10 13:23:51,916 INFO [org.ovirt.engine.core.bll.InstallVdsInternalCommand] (org.ovirt.thread.pool-6-thread-49) [5d003ac8] Lock Acquired to object EngineLock [exclusiveLocks= key: f689c28a-07de-4408-ac86-81382b6dd288 value: VDS >, sharedLocks= ] >2014-09-10 13:23:51,931 INFO [org.ovirt.engine.core.bll.InstallVdsInternalCommand] (org.ovirt.thread.pool-6-thread-49) [5d003ac8] Running command: InstallVdsInternalCommand internal: true. Entities affected : ID: f689c28a-07de-4408-ac86-81382b6dd288 Type: VDS >2014-09-10 13:23:51,931 INFO [org.ovirt.engine.core.bll.InstallVdsInternalCommand] (org.ovirt.thread.pool-6-thread-49) [5d003ac8] Before Installation host f689c28a-07de-4408-ac86-81382b6dd288, rack >2014-09-10 13:23:51,936 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-6-thread-49) [5d003ac8] START, SetVdsStatusVDSCommand(HostName = rack, HostId = f689c28a-07de-4408-ac86-81382b6dd288, status=Installing, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 75aee44b >2014-09-10 13:23:51,956 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-6-thread-49) [5d003ac8] FINISH, SetVdsStatusVDSCommand, log id: 75aee44b >2014-09-10 13:23:51,983 INFO [org.ovirt.engine.core.bll.InstallerMessages] (org.ovirt.thread.pool-6-thread-49) [5d003ac8] Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Connected to host dell-r210ii-16.rhev.lab.eng.brq.redhat.com with SSH key fingerprint: 25:12:7b:de:30:1d:b3:5e:bf:d2:a6:c7:37:77:d3:b1 >2014-09-10 13:23:51,995 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-49) [5d003ac8] Correlation ID: 5d003ac8, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Connected to host dell-r210ii-16.rhev.lab.eng.brq.redhat.com with SSH key fingerprint: 25:12:7b:de:30:1d:b3:5e:bf:d2:a6:c7:37:77:d3:b1. >2014-09-10 13:23:52,062 INFO [org.ovirt.engine.core.bll.VdsDeploy] (org.ovirt.thread.pool-6-thread-49) [5d003ac8] Installation of dell-r210ii-16.rhev.lab.eng.brq.redhat.com. Executing command via SSH umask 0077; MYTMP="$(TMPDIR="${OVIRT_TMPDIR}" mktemp -t ovirt-XXXXXXXXXX)"; trap "chmod -R u+rwX \"${MYTMP}\" > /dev/null 2>&1; rm -fr \"${MYTMP}\" > /dev/null 2>&1" 0; rm -fr "${MYTMP}" && mkdir "${MYTMP}" && tar --warning=no-timestamp -C "${MYTMP}" -x && "${MYTMP}"/setup DIALOG/dialect=str:machine DIALOG/customization=bool:True < /var/cache/ovirt-engine/ovirt-host-deploy.tar >2014-09-10 13:23:52,076 INFO [org.ovirt.engine.core.uutils.ssh.SSHDialog] (org.ovirt.thread.pool-6-thread-49) SSH execute root@dell-r210ii-16.rhev.lab.eng.brq.redhat.com 'umask 0077; MYTMP="$(TMPDIR="${OVIRT_TMPDIR}" mktemp -t ovirt-XXXXXXXXXX)"; trap "chmod -R u+rwX \"${MYTMP}\" > /dev/null 2>&1; rm -fr \"${MYTMP}\" > /dev/null 2>&1" 0; rm -fr "${MYTMP}" && mkdir "${MYTMP}" && tar --warning=no-timestamp -C "${MYTMP}" -x && "${MYTMP}"/setup DIALOG/dialect=str:machine DIALOG/customization=bool:True' >2014-09-10 13:23:52,849 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Stage: Initializing >2014-09-10 13:23:52,859 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 5d003ac8, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Stage: Initializing. >2014-09-10 13:23:52,871 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Stage: Environment setup >2014-09-10 13:23:52,881 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 5d003ac8, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Stage: Environment setup. >2014-09-10 13:23:52,881 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Stage: Environment packages setup >2014-09-10 13:23:52,892 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 5d003ac8, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Stage: Environment packages setup. >2014-09-10 13:23:54,869 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-12) Initializing Host: rack >2014-09-10 13:23:58,115 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Stage: Programs detection >2014-09-10 13:23:58,127 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 5d003ac8, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Stage: Programs detection. >2014-09-10 13:23:58,127 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Stage: Environment customization >2014-09-10 13:23:58,138 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 5d003ac8, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Stage: Environment customization. >2014-09-10 13:23:58,512 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Kdump supported >2014-09-10 13:23:58,521 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 5d003ac8, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Kdump supported. >2014-09-10 13:23:58,534 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Logs at host located at: '/tmp/ovirt-host-deploy-20140910132351-ohur1q.log' >2014-09-10 13:23:58,545 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 5d003ac8, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Logs at host located at: '/tmp/ovirt-host-deploy-20140910132351-ohur1q.log'. >2014-09-10 13:23:58,637 INFO [org.ovirt.engine.core.bll.VdsDeploy] (VdsDeploy) Host dell-r210ii-16.rhev.lab.eng.brq.redhat.com reports unique id 4C4C4544-0034-5310-8052-B4C04F4A354A >2014-09-10 13:23:58,640 INFO [org.ovirt.engine.core.bll.VdsDeploy] (VdsDeploy) Assigning unique id 4C4C4544-0034-5310-8052-B4C04F4A354A to Host dell-r210ii-16.rhev.lab.eng.brq.redhat.com >2014-09-10 13:23:58,731 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Disabling Kdump integration >2014-09-10 13:23:58,748 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 5d003ac8, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Disabling Kdump integration. >2014-09-10 13:23:58,768 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Stage: Setup validation >2014-09-10 13:23:58,778 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 5d003ac8, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Stage: Setup validation. >2014-09-10 13:23:58,807 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Hardware supports virtualization >2014-09-10 13:23:58,817 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 5d003ac8, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Hardware supports virtualization. >2014-09-10 13:23:59,047 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Stage: Transaction setup >2014-09-10 13:23:59,057 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 5d003ac8, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Stage: Transaction setup. >2014-09-10 13:23:59,099 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Stage: Misc configuration >2014-09-10 13:23:59,109 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 5d003ac8, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Stage: Misc configuration. >2014-09-10 13:23:59,109 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Stage: Package installation >2014-09-10 13:23:59,117 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 5d003ac8, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Stage: Package installation. >2014-09-10 13:24:06,367 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Stage: Misc configuration >2014-09-10 13:24:06,379 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 5d003ac8, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Stage: Misc configuration. >2014-09-10 13:24:06,379 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Setting time >2014-09-10 13:24:06,390 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 5d003ac8, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Setting time. >2014-09-10 13:24:07,985 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Enrolling certificate >2014-09-10 13:24:07,998 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 5d003ac8, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Enrolling certificate. >2014-09-10 13:24:09,212 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Stage: Transaction commit >2014-09-10 13:24:09,222 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 5d003ac8, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Stage: Transaction commit. >2014-09-10 13:24:09,286 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Stage: Closing up >2014-09-10 13:24:09,297 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 5d003ac8, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Stage: Closing up. >2014-09-10 13:24:10,064 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Stopping libvirtd >2014-09-10 13:24:10,076 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 5d003ac8, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Stopping libvirtd. >2014-09-10 13:24:10,327 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Starting vdsm >2014-09-10 13:24:10,337 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 5d003ac8, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Starting vdsm. >2014-09-10 13:24:12,133 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Stage: Pre-termination >2014-09-10 13:24:12,145 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 5d003ac8, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Stage: Pre-termination. >2014-09-10 13:24:12,162 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Retrieving installation logs to: '/var/log/ovirt-engine/host-deploy/ovirt-20140910132412-dell-r210ii-16.rhev.lab.eng.brq.redhat.com-5d003ac8.log' >2014-09-10 13:24:12,176 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 5d003ac8, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Retrieving installation logs to: '/var/log/ovirt-engine/host-deploy/ovirt-20140910132412-dell-r210ii-16.rhev.lab.eng.brq.redhat.com-5d003ac8.log'. >2014-09-10 13:24:12,407 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation dell-r210ii-16.rhev.lab.eng.brq.redhat.com: Stage: Termination >2014-09-10 13:24:12,417 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 5d003ac8, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Stage: Termination. >2014-09-10 13:24:12,501 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to dell-r210ii-16.rhev.lab.eng.brq.redhat.com/10.34.62.208 >2014-09-10 13:24:12,502 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-10 13:24:12,505 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-10 13:24:12,506 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (org.ovirt.thread.pool-6-thread-48) Exception during connection: java.util.concurrent.ExecutionException: java.net.ConnectException: Connection refused > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:94) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:98) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:56) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:57) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer$2.call(JsonRpcVdsServer.java:871) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer$2.call(JsonRpcVdsServer.java:867) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_65] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_65] > at java.lang.Thread.run(Thread.java:745) [rt.jar:1.7.0_65] >Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:465) [rt.jar:1.7.0_65] > at sun.nio.ch.Net.connect(Net.java:457) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:87) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:77) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-09-10 13:24:12,517 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.PollVDSCommand] (org.ovirt.thread.pool-6-thread-49) [5d003ac8] java.util.concurrent.ExecutionException: org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcRunTimeException: Connection issues during send request >2014-09-10 13:24:12,520 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.PollVDSCommand] (org.ovirt.thread.pool-6-thread-49) [5d003ac8] Command PollVDSCommand(HostName = rack, HostId = f689c28a-07de-4408-ac86-81382b6dd288) execution failed. Exception: RuntimeException: java.util.concurrent.ExecutionException: org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcRunTimeException: Connection issues during send request >2014-09-10 13:24:13,021 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to dell-r210ii-16.rhev.lab.eng.brq.redhat.com/10.34.62.208 >2014-09-10 13:24:13,105 INFO [org.ovirt.engine.core.bll.network.NetworkConfigurator] (org.ovirt.thread.pool-6-thread-49) [5d003ac8] Engine managed to communicate with VDSM agent on host rack >2014-09-10 13:24:13,433 INFO [org.ovirt.engine.core.bll.network.NetworkConfigurator] (org.ovirt.thread.pool-6-thread-49) [5d003ac8] The management network rhevm is already configured on host rack >2014-09-10 13:24:13,436 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-6-thread-49) [5d003ac8] START, SetVdsStatusVDSCommand(HostName = rack, HostId = f689c28a-07de-4408-ac86-81382b6dd288, status=Initializing, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 60d8558a >2014-09-10 13:24:13,455 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-6-thread-49) [5d003ac8] FINISH, SetVdsStatusVDSCommand, log id: 60d8558a >2014-09-10 13:24:13,455 INFO [org.ovirt.engine.core.bll.InstallVdsInternalCommand] (org.ovirt.thread.pool-6-thread-49) [5d003ac8] After Installation host rack, VDS >2014-09-10 13:24:13,470 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-49) [5d003ac8] Correlation ID: 5d003ac8, Job ID: f04c0777-cd59-4cea-8771-d9946f27e6e2, Call Stack: null, Custom Event ID: -1, Message: Host rack installed >2014-09-10 13:24:13,481 INFO [org.ovirt.engine.core.bll.InstallVdsInternalCommand] (org.ovirt.thread.pool-6-thread-49) [5d003ac8] Lock freed to object EngineLock [exclusiveLocks= key: f689c28a-07de-4408-ac86-81382b6dd288 value: VDS >, sharedLocks= ] >2014-09-10 13:24:15,946 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-64) START, GetHardwareInfoVDSCommand(HostName = rack, HostId = f689c28a-07de-4408-ac86-81382b6dd288, vds=Host[rack,f689c28a-07de-4408-ac86-81382b6dd288]), log id: 57bc2aa8 >2014-09-10 13:24:15,978 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-64) FINISH, GetHardwareInfoVDSCommand, log id: 57bc2aa8 >2014-09-10 13:24:16,031 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-64) [488773a4] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: f689c28a-07de-4408-ac86-81382b6dd288 Type: VDS >2014-09-10 13:24:16,094 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-64) [488773a4] START, GetHardwareInfoVDSCommand(HostName = rack, HostId = f689c28a-07de-4408-ac86-81382b6dd288, vds=Host[rack,f689c28a-07de-4408-ac86-81382b6dd288]), log id: 13f9a5c4 >2014-09-10 13:24:16,101 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-64) [488773a4] FINISH, GetHardwareInfoVDSCommand, log id: 13f9a5c4 >2014-09-10 13:24:16,151 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-64) [3d96d9a1] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001ad Type: StoragePool >2014-09-10 13:24:16,167 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-64) [3d96d9a1] START, SetMOMPolicyParametersVDSCommand(HostName = rack, HostId = f689c28a-07de-4408-ac86-81382b6dd288), log id: 7cfa360f >2014-09-10 13:24:16,225 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-64) [3d96d9a1] FINISH, SetMOMPolicyParametersVDSCommand, log id: 7cfa360f >2014-09-10 13:24:16,240 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-64) [3d96d9a1] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host rack was set to Up. >2014-09-10 13:24:16,251 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-64) [3d96d9a1] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host rack. >2014-09-10 13:24:16,282 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-64) [762ceea8] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: f689c28a-07de-4408-ac86-81382b6dd288 Type: VDS >2014-09-10 13:24:16,290 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-64) [29539bf3] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: f689c28a-07de-4408-ac86-81382b6dd288 Type: VDS >2014-09-10 13:59:14,659 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-4) Principal admin::internal synchronized >2014-09-10 13:59:15,181 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-10) Attempting to update VMs/Templates Ovf. >2014-09-10 14:11:03,803 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp-/127.0.0.1:8702-5) [37e7e802] Running command: LogoutUserCommand internal: false. >2014-09-10 14:11:03,846 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp-/127.0.0.1:8702-10) [295dc995] Running command: LogoutUserCommand internal: false. >2014-09-10 14:11:03,852 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp-/127.0.0.1:8702-4) [757a28c7] Running command: LogoutUserCommand internal: false. >2014-09-10 14:11:03,856 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-5) [37e7e802] Correlation ID: 37e7e802, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-09-10 14:11:03,851 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp-/127.0.0.1:8702-6) [e16b7c2] Running command: LogoutUserCommand internal: false. >2014-09-10 14:11:03,859 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp-/127.0.0.1:8702-7) [2ac5a46a] Running command: LogoutUserCommand internal: false. >2014-09-10 14:11:03,888 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp-/127.0.0.1:8702-1) [2d2da368] Running command: LogoutUserCommand internal: false. >2014-09-10 14:11:03,890 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-6) [e16b7c2] Correlation ID: e16b7c2, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-09-10 14:11:03,898 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-4) [757a28c7] Correlation ID: 757a28c7, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-09-10 14:11:03,917 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-10) [295dc995] Correlation ID: 295dc995, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-09-10 14:11:03,921 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-7) [2ac5a46a] Correlation ID: 2ac5a46a, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-09-10 14:11:03,931 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-1) [2d2da368] Correlation ID: 2d2da368, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-09-10 14:11:04,527 INFO [org.ovirt.engine.core.utils.servlet.UnsupportedLocaleHelper] (ajp-/127.0.0.1:8702-3) Invalid locale found in configuration: >2014-09-10 14:11:04,527 INFO [org.ovirt.engine.core.utils.servlet.UnsupportedLocaleHelper] (ajp-/127.0.0.1:8702-3) Invalid locale found in configuration: >2014-09-10 14:11:51,966 INFO [org.ovirt.engine.core.bll.aaa.LoginAdminUserCommand] (ajp-/127.0.0.1:8702-9) Running command: LoginAdminUserCommand internal: false. >2014-09-10 14:11:51,977 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-9) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-09-10 14:13:44,829 INFO [org.ovirt.engine.core.bll.MaintenanceNumberOfVdssCommand] (org.ovirt.thread.pool-6-thread-31) [655aab1e] Running command: MaintenanceNumberOfVdssCommand internal: false. Entities affected : ID: f689c28a-07de-4408-ac86-81382b6dd288 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2014-09-10 14:13:44,843 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-6-thread-31) [655aab1e] START, SetVdsStatusVDSCommand(HostName = rack, HostId = f689c28a-07de-4408-ac86-81382b6dd288, status=PreparingForMaintenance, nonOperationalReason=NONE, stopSpmFailureLogged=true), log id: 5b483271 >2014-09-10 14:13:44,861 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-6-thread-31) [655aab1e] FINISH, SetVdsStatusVDSCommand, log id: 5b483271 >2014-09-10 14:13:44,956 INFO [org.ovirt.engine.core.bll.MaintenanceVdsCommand] (org.ovirt.thread.pool-6-thread-31) [655aab1e] Running command: MaintenanceVdsCommand internal: true. Entities affected : ID: f689c28a-07de-4408-ac86-81382b6dd288 Type: VDS >2014-09-10 14:13:44,971 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-31) [655aab1e] Correlation ID: 655aab1e, Job ID: a757f9e0-ce1b-4af0-b821-875ea8d06b70, Call Stack: null, Custom Event ID: -1, Message: Host rack was switched to Maintenance mode by admin. >2014-09-10 14:13:45,519 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-76) Updated vds status from Preparing for Maintenance to Maintenance in database, vds = f689c28a-07de-4408-ac86-81382b6dd288 : rack >2014-09-10 14:13:45,553 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-6-thread-33) Clearing cache of pool: 00000002-0002-0002-0002-0000000001ad for problematic entities of VDS: rack. >2014-09-10 14:13:45,554 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-6-thread-33) Removing vds [f689c28a-07de-4408-ac86-81382b6dd288] from the domain in maintenance cache >2014-09-10 14:13:45,554 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-6-thread-33) Removing host(s) [f689c28a-07de-4408-ac86-81382b6dd288] from hosts unseen domain report cache >2014-09-10 14:14:12,251 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (ajp-/127.0.0.1:8702-5) [7f7de479] Lock Acquired to object EngineLock [exclusiveLocks= key: f689c28a-07de-4408-ac86-81382b6dd288 value: VDS >, sharedLocks= ] >2014-09-10 14:14:12,286 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (org.ovirt.thread.pool-6-thread-35) [7f7de479] Running command: ActivateVdsCommand internal: false. Entities affected : ID: f689c28a-07de-4408-ac86-81382b6dd288 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2014-09-10 14:14:12,287 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (org.ovirt.thread.pool-6-thread-35) [7f7de479] Before acquiring lock in order to prevent monitoring for host rack from data-center Default >2014-09-10 14:14:12,287 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (org.ovirt.thread.pool-6-thread-35) [7f7de479] Lock acquired, from now a monitoring of host will be skipped for host rack from data-center Default >2014-09-10 14:14:12,301 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-6-thread-35) [7f7de479] START, SetVdsStatusVDSCommand(HostName = rack, HostId = f689c28a-07de-4408-ac86-81382b6dd288, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 3465ccf5 >2014-09-10 14:14:12,316 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-6-thread-35) [7f7de479] FINISH, SetVdsStatusVDSCommand, log id: 3465ccf5 >2014-09-10 14:14:12,327 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (org.ovirt.thread.pool-6-thread-35) [7f7de479] Activate finished. Lock released. Monitoring can run now for host rack from data-center Default >2014-09-10 14:14:12,337 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-35) [7f7de479] Correlation ID: 7f7de479, Job ID: 96539c7c-b0f9-4d55-8e62-71dd6aa9e2ba, Call Stack: null, Custom Event ID: -1, Message: Host rack was activated by admin. >2014-09-10 14:14:12,347 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (org.ovirt.thread.pool-6-thread-35) [7f7de479] Lock freed to object EngineLock [exclusiveLocks= key: f689c28a-07de-4408-ac86-81382b6dd288 value: VDS >, sharedLocks= ] >2014-09-10 14:14:12,653 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-30) START, GetHardwareInfoVDSCommand(HostName = rack, HostId = f689c28a-07de-4408-ac86-81382b6dd288, vds=Host[rack,f689c28a-07de-4408-ac86-81382b6dd288]), log id: 214c5d7a >2014-09-10 14:14:12,665 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-30) FINISH, GetHardwareInfoVDSCommand, log id: 214c5d7a >2014-09-10 14:14:12,730 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-30) [71b56914] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: f689c28a-07de-4408-ac86-81382b6dd288 Type: VDS >2014-09-10 14:14:12,743 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-30) [7cadb7b2] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001ad Type: StoragePool >2014-09-10 14:14:12,762 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-30) [7cadb7b2] START, SetMOMPolicyParametersVDSCommand(HostName = rack, HostId = f689c28a-07de-4408-ac86-81382b6dd288), log id: 3e156850 >2014-09-10 14:14:12,816 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-30) [7cadb7b2] FINISH, SetMOMPolicyParametersVDSCommand, log id: 3e156850 >2014-09-10 14:14:12,834 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-30) [7cadb7b2] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host rack was set to Up. >2014-09-10 14:14:12,847 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-30) [7cadb7b2] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host rack. >2014-09-10 14:14:12,878 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-30) [93c1504] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: f689c28a-07de-4408-ac86-81382b6dd288 Type: VDS
You cannot view the attachment while viewing its details because your browser does not support IFRAMEs.
View the attachment on a separate page
.
View Attachment As Raw
Actions:
View
Attachments on
bug 1140175
: 936121 |
936122