Login
[x]
Log in using an account from:
Fedora Account System
Red Hat Associate
Red Hat Customer
Or login using a Red Hat Bugzilla account
Forgot Password
Login:
Hide Forgot
Create an Account
Red Hat Bugzilla – Attachment 925779 Details for
Bug 1128778
unable to activate master domain after switching to jsonrpc
[?]
New
Simple Search
Advanced Search
My Links
Browse
Requests
Reports
Current State
Search
Tabular reports
Graphical reports
Duplicates
Other Reports
User Changes
Plotly Reports
Bug Status
Bug Severity
Non-Defaults
|
Product Dashboard
Help
Page Help!
Bug Writing Guidelines
What's new
Browser Support Policy
5.0.4.rh83 Release notes
FAQ
Guides index
User guide
Web Services
Contact
Legal
This site requires JavaScript to be enabled to function correctly, please enable it.
engine log
engine.log (text/x-log), 3.41 MB, created by
Petr Beňas
on 2014-08-11 14:23:17 UTC
(
hide
)
Description:
engine log
Filename:
MIME Type:
Creator:
Petr Beňas
Created:
2014-08-11 14:23:17 UTC
Size:
3.41 MB
patch
obsolete
>2014-07-22 17:03:26,049 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Loaded file "/usr/share/ovirt-engine/services/ovirt-engine/ovirt-engine.conf". >2014-07-22 17:03:26,050 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) The file "/etc/ovirt-engine/engine.conf" doesn't exist or isn't readable. Will return an empty set of properties. >2014-07-22 17:03:26,050 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Loaded file "/etc/ovirt-engine/engine.conf.d/10-setup-database.conf". >2014-07-22 17:03:26,051 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Loaded file "/etc/ovirt-engine/engine.conf.d/10-setup-jboss.conf". >2014-07-22 17:03:26,052 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Loaded file "/etc/ovirt-engine/engine.conf.d/10-setup-pki.conf". >2014-07-22 17:03:26,054 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Loaded file "/etc/ovirt-engine/engine.conf.d/10-setup-protocols.conf". >2014-07-22 17:03:26,055 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_AJP_ENABLED" is "true". >2014-07-22 17:03:26,065 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_AJP_PORT" is "8702". >2014-07-22 17:03:26,066 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_APPS" is "engine.ear restapi.war legacy_restapi.war". >2014-07-22 17:03:26,066 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_CACHE" is "/var/cache/ovirt-engine". >2014-07-22 17:03:26,067 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_DB_CHECK_INTERVAL" is "1000". >2014-07-22 17:03:26,067 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_DB_CONNECTION_TIMEOUT" is "300000". >2014-07-22 17:03:26,067 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_DB_DATABASE" is "engine". >2014-07-22 17:03:26,068 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_DB_DRIVER" is "org.postgresql.Driver". >2014-07-22 17:03:26,068 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_DB_HOST" is "localhost". >2014-07-22 17:03:26,071 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_DB_MAX_CONNECTIONS" is "100". >2014-07-22 17:03:26,072 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_DB_MIN_CONNECTIONS" is "1". >2014-07-22 17:03:26,072 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_DB_PASSWORD" is "***". >2014-07-22 17:03:26,072 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_DB_PORT" is "5432". >2014-07-22 17:03:26,073 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_DB_SECURED" is "False". >2014-07-22 17:03:26,073 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_DB_SECURED_VALIDATION" is "False". >2014-07-22 17:03:26,074 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_DB_URL" is "jdbc:postgresql://localhost:5432/engine?sslfactory=org.postgresql.ssl.NonValidatingFactory". >2014-07-22 17:03:26,074 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_DB_USER" is "engine". >2014-07-22 17:03:26,090 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_DEBUG_ADDRESS" is "". >2014-07-22 17:03:26,090 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_DOC" is "/usr/share/doc/ovirt-engine". >2014-07-22 17:03:26,091 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_ETC" is "/etc/ovirt-engine". >2014-07-22 17:03:26,091 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_EXTENSION_PATH" is "/usr/share/ovirt-engine/extensions.d:/etc/ovirt-engine/extensions.d". >2014-07-22 17:03:26,092 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_FQDN" is "pb-rh35.rhev.lab.eng.brq.redhat.com". >2014-07-22 17:03:26,092 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_GROUP" is "ovirt". >2014-07-22 17:03:26,093 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_HEAP_MAX" is "1g". >2014-07-22 17:03:26,093 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_HEAP_MIN" is "1g". >2014-07-22 17:03:26,094 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_HTTPS_ENABLED" is "false". >2014-07-22 17:03:26,094 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_HTTPS_PORT" is "None". >2014-07-22 17:03:26,103 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_HTTPS_PROTOCOLS" is "SSLv3,TLSv1,TLSv1.1,TLSv1.2". >2014-07-22 17:03:26,103 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_HTTP_ENABLED" is "false". >2014-07-22 17:03:26,104 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_HTTP_PORT" is "None". >2014-07-22 17:03:26,104 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_JAVA_MODULEPATH" is "/usr/share/ovirt-engine/modules". >2014-07-22 17:03:26,105 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_JVM_ARGS" is " -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath="/var/log/ovirt-engine/dump"". >2014-07-22 17:03:26,105 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_LOG" is "/var/log/ovirt-engine". >2014-07-22 17:03:26,106 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_LOG_TO_CONSOLE" is "false". >2014-07-22 17:03:26,106 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_MANUAL" is "/usr/share/ovirt-engine/manual". >2014-07-22 17:03:26,115 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_PERM_MAX" is "256m". >2014-07-22 17:03:26,116 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_PERM_MIN" is "256m". >2014-07-22 17:03:26,116 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_PKI" is "/etc/pki/ovirt-engine". >2014-07-22 17:03:26,116 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_PKI_CA" is "/etc/pki/ovirt-engine/ca.pem". >2014-07-22 17:03:26,117 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_PKI_ENGINE_CERT" is "/etc/pki/ovirt-engine/certs/engine.cer". >2014-07-22 17:03:26,118 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_PKI_ENGINE_STORE" is "/etc/pki/ovirt-engine/keys/engine.p12". >2014-07-22 17:03:26,118 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_PKI_ENGINE_STORE_ALIAS" is "1". >2014-07-22 17:03:26,126 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_PKI_ENGINE_STORE_PASSWORD" is "***". >2014-07-22 17:03:26,126 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_PKI_TRUST_STORE" is "/etc/pki/ovirt-engine/.truststore". >2014-07-22 17:03:26,127 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_PKI_TRUST_STORE_PASSWORD" is "***". >2014-07-22 17:03:26,128 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_PROPERTIES" is " jsse.enableSNIExtension=false "java.security.krb5.conf=/etc/ovirt-engine/krb5.conf"". >2014-07-22 17:03:26,128 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_PROXY_ENABLED" is "true". >2014-07-22 17:03:26,129 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_PROXY_HTTPS_PORT" is "443". >2014-07-22 17:03:26,129 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_PROXY_HTTP_PORT" is "80". >2014-07-22 17:03:26,130 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_REPORTS_UI" is "/var/lib/ovirt-engine/reports.xml". >2014-07-22 17:03:26,130 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_STOP_INTERVAL" is "1". >2014-07-22 17:03:26,130 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_STOP_TIME" is "10". >2014-07-22 17:03:26,136 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_TMP" is "/var/tmp/ovirt-engine". >2014-07-22 17:03:26,136 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_UP_MARK" is "/var/lib/ovirt-engine/engine.up". >2014-07-22 17:03:26,137 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_URI" is "/ovirt-engine". >2014-07-22 17:03:26,137 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_USER" is "ovirt". >2014-07-22 17:03:26,138 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_USR" is "/usr/share/ovirt-engine". >2014-07-22 17:03:26,138 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_VAR" is "/var/lib/ovirt-engine". >2014-07-22 17:03:26,139 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "ENGINE_VERBOSE_GC" is "false". >2014-07-22 17:03:26,139 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "JBOSS_HOME" is "/usr/share/ovirt-engine-jboss-as". >2014-07-22 17:03:26,146 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "JBOSS_RUNTIME" is "/var/lib/ovirt-engine/jboss_runtime". >2014-07-22 17:03:26,146 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-1) Value of property "SENSITIVE_KEYS" is ",ENGINE_DB_PASSWORD,ENGINE_PKI_TRUST_STORE_PASSWORD,ENGINE_PKI_ENGINE_STORE_PASSWORD". >2014-07-22 17:03:26,950 INFO [org.ovirt.engine.core.bll.Backend] (MSC service thread 1-2) Start initializing Backend >2014-07-22 17:03:27,568 INFO [org.ovirt.engine.core.utils.osinfo.OsInfoPreferencesLoader] (MSC service thread 1-2) Loaded file /etc/ovirt-engine/osinfo.conf.d/00-defaults.properties >2014-07-22 17:03:27,691 INFO [org.ovirt.engine.core.bll.Backend] (MSC service thread 1-2) Running ovirt-engine 3.5.0-0.0.master.20140715172116.git4687dc1.el6 >2014-07-22 17:03:27,692 INFO [org.ovirt.engine.core.bll.CpuFlagsManagerHandler] (MSC service thread 1-2) Start initializing dictionaries >2014-07-22 17:03:27,705 INFO [org.ovirt.engine.core.bll.CpuFlagsManagerHandler] (MSC service thread 1-2) Finished initializing dictionaries >2014-07-22 17:03:27,706 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (MSC service thread 1-2) Start initializing AuditLogCleanupManager >2014-07-22 17:03:27,707 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (MSC service thread 1-2) Setting audit cleanup manager to run at: 35 35 3 * * ? >2014-07-22 17:03:27,748 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (MSC service thread 1-2) Finished initializing AuditLogCleanupManager >2014-07-22 17:03:27,748 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (MSC service thread 1-2) Start initializing CommandEntityCleanupManager >2014-07-22 17:03:27,749 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (MSC service thread 1-2) Setting command entity cleanup manager to run at: 35 35 3 * * ? >2014-07-22 17:03:27,750 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (MSC service thread 1-2) Finished initializing CommandEntityCleanupManager >2014-07-22 17:03:27,753 INFO [org.ovirt.engine.core.bll.TagsDirector] (MSC service thread 1-2) Start initializing TagsDirector >2014-07-22 17:03:27,780 INFO [org.ovirt.engine.core.bll.TagsDirector] (MSC service thread 1-2) Tag root added to tree >2014-07-22 17:03:27,797 INFO [org.ovirt.engine.core.bll.TagsDirector] (MSC service thread 1-2) Finished initializing TagsDirector >2014-07-22 17:03:27,800 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (MSC service thread 1-2) Start initializing IsoDomainListSyncronizer >2014-07-22 17:03:27,815 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (MSC service thread 1-2) Finished initializing IsoDomainListSyncronizer >2014-07-22 17:03:27,971 INFO [org.ovirt.engine.core.bll.Backend] (MSC service thread 1-2) Completed initializing handlers >2014-07-22 17:03:28,003 INFO [org.ovirt.engine.core.utils.ErrorTranslatorImpl] (MSC service thread 1-2) Start initializing ErrorTranslatorImpl >2014-07-22 17:03:28,023 WARN [org.ovirt.engine.core.utils.ErrorTranslatorImpl] (MSC service thread 1-2) Code MAC_ADDRESS_IS_IN_USE appears more than once in string table. >2014-07-22 17:03:28,024 INFO [org.ovirt.engine.core.utils.ErrorTranslatorImpl] (MSC service thread 1-2) Finished initializing ErrorTranslatorImpl >2014-07-22 17:03:28,025 INFO [org.ovirt.engine.core.utils.ErrorTranslatorImpl] (MSC service thread 1-2) Start initializing ErrorTranslatorImpl >2014-07-22 17:03:28,026 INFO [org.ovirt.engine.core.utils.ErrorTranslatorImpl] (MSC service thread 1-2) Finished initializing ErrorTranslatorImpl >2014-07-22 17:03:28,026 INFO [org.ovirt.engine.core.bll.Backend] (MSC service thread 1-2) Mark incomplete jobs as UNKNOWN >2014-07-22 17:03:28,071 INFO [org.ovirt.engine.core.bll.job.JobRepositoryCleanupManager] (MSC service thread 1-2) Start initializing JobRepositoryCleanupManager >2014-07-22 17:03:28,072 INFO [org.ovirt.engine.core.bll.job.JobRepositoryCleanupManager] (MSC service thread 1-2) Finished initializing JobRepositoryCleanupManager >2014-07-22 17:03:28,073 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (MSC service thread 1-2) Start initializing AutoRecoveryManager >2014-07-22 17:03:28,076 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (MSC service thread 1-2) Finished initializing AutoRecoveryManager >2014-07-22 17:03:28,077 INFO [org.ovirt.engine.core.dal.job.ExecutionMessageDirector] (MSC service thread 1-2) Start initializing ExecutionMessageDirector >2014-07-22 17:03:28,081 INFO [org.ovirt.engine.core.dal.job.ExecutionMessageDirector] (MSC service thread 1-2) Finished initializing ExecutionMessageDirector >2014-07-22 17:03:28,143 INFO [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (MSC service thread 1-2) Start of enabled extensions list >2014-07-22 17:03:28,147 INFO [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (MSC service thread 1-2) Instance name: 'builtin-authn-internal', Extension name: 'Internal Authn (Built-in)', Version: 'N/A', Notes: '', License: 'ASL 2.0', Home: 'http://www.ovirt.org', Author 'The oVirt Project', Build interface Version: '0', File: 'N/A', Initialized: 'true' >2014-07-22 17:03:28,148 INFO [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (MSC service thread 1-2) Instance name: 'internal', Extension name: 'Internal Authz (Built-in)', Version: 'N/A', Notes: '', License: 'ASL 2.0', Home: 'http://www.ovirt.org', Author 'The oVirt Project', Build interface Version: '0', File: 'N/A', Initialized: 'true' >2014-07-22 17:03:28,148 INFO [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (MSC service thread 1-2) End of enabled extensions list >2014-07-22 17:03:28,257 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (MSC service thread 1-3) Start initializing DbUserCacheManager >2014-07-22 17:03:28,264 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (MSC service thread 1-3) Finished initializing DbUserCacheManager >2014-07-22 17:03:28,288 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (MSC service thread 1-3) Initialization of AsyncTaskManager completed successfully. >2014-07-22 17:03:28,296 INFO [org.ovirt.engine.core.vdsbroker.ResourceManager] (MSC service thread 1-3) Start initializing ResourceManager >2014-07-22 17:03:28,318 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-1) The user admin from authz extension internal got updated since last interval >2014-07-22 17:03:28,418 INFO [org.ovirt.engine.core.vdsbroker.ResourceManager] (MSC service thread 1-3) Finished initializing ResourceManager >2014-07-22 17:03:28,424 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (MSC service thread 1-3) Initialization of OvfDataUpdater completed successfully. >2014-07-22 17:03:28,424 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (MSC service thread 1-3) Start scheduling to enable vds load balancer >2014-07-22 17:03:28,425 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (MSC service thread 1-3) Finished scheduling to enable vds load balancer >2014-07-22 17:03:28,425 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (MSC service thread 1-3) Start HA Reservation check >2014-07-22 17:03:28,428 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (MSC service thread 1-3) Finished HA Reservation check >2014-07-22 17:03:28,438 INFO [org.ovirt.engine.core.bll.network.MacPoolManagerRanges] (org.ovirt.thread.pool-8-thread-1) Start initializing MacPoolManagerRanges >2014-07-22 17:03:28,438 INFO [org.ovirt.engine.core.bll.InitBackendServicesOnStartupBean] (MSC service thread 1-3) Init VM custom properties utilities >2014-07-22 17:03:28,440 INFO [org.ovirt.engine.core.bll.InitBackendServicesOnStartupBean] (MSC service thread 1-3) Init device custom properties utilities >2014-07-22 17:03:28,445 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (MSC service thread 1-3) Initializing Scheduling manager >2014-07-22 17:03:28,476 INFO [org.ovirt.engine.core.bll.network.MacPoolManagerRanges] (org.ovirt.thread.pool-8-thread-1) Finished initializing. Available MACs in pool: 255 >2014-07-22 17:03:28,521 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (MSC service thread 1-3) External scheduler disabled, discovery skipped >2014-07-22 17:03:28,521 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (MSC service thread 1-3) Initialized Scheduling manager >2014-07-22 17:03:28,522 INFO [org.ovirt.engine.core.bll.dwh.DwhHeartBeat] (MSC service thread 1-3) Initializing DWH Heart Beat >2014-07-22 17:03:28,527 INFO [org.ovirt.engine.core.bll.dwh.DwhHeartBeat] (MSC service thread 1-3) DWH Heart Beat initialized >2014-07-22 17:03:59,478 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp--127.0.0.1-8702-1) Running command: LoginUserCommand internal: false. >2014-07-22 17:03:59,487 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-1) Cannot convert the string Host ${VdsName} is not responding. It will stay in Connecting state for a grace period of ${Seconds} seconds and after that an attempt to fence the host will be issued. to AuditLogType, the key does not exist in the AuditLogType declared types >2014-07-22 17:03:59,494 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-1) AuditLogType: UNASSIGNED not exist in string table >2014-07-22 17:03:59,495 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-1) AuditLogType: VDS_AUTO_FENCE_STATUS not exist in string table >2014-07-22 17:03:59,496 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-1) AuditLogType: VDS_AUTO_FENCE_STATUS_FAILED not exist in string table >2014-07-22 17:03:59,498 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-1) AuditLogType: VDS_AUTO_FENCE_FAILED_CALL_FENCE_SPM not exist in string table >2014-07-22 17:03:59,499 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-1) AuditLogType: VDS_HIGH_NETWORK_USE not exist in string table >2014-07-22 17:03:59,500 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-1) AuditLogType: USER_FAILED_REMOVE_VM not exist in string table >2014-07-22 17:03:59,501 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-1) AuditLogType: USER_RUN_UNLOCK_ENTITY_SCRIPT not exist in string table >2014-07-22 17:03:59,503 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-1) AuditLogType: VDS_NETWORK_MTU_DIFFER_FROM_LOGICAL_NETWORK not exist in string table >2014-07-22 17:03:59,504 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-1) AuditLogType: VDS_HOST_IN_CONNECTING_STATE not exist in string table >2014-07-22 17:03:59,505 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-1) AuditLogType: STORAGE_ACTIVATE_ASYNC not exist in string table >2014-07-22 17:03:59,513 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-1) AuditLogType: DWH_STOPPED not exist in string table >2014-07-22 17:03:59,514 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-1) AuditLogType: DWH_STARTED not exist in string table >2014-07-22 17:03:59,515 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-1) AuditLogType: DWH_ERROR not exist in string table >2014-07-22 17:03:59,516 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-1) AuditLogType: USER_REMOVE_AUDIT_LOG not exist in string table >2014-07-22 17:03:59,517 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-1) AuditLogType: USER_REMOVE_AUDIT_LOG_FAILED not exist in string table >2014-07-22 17:03:59,518 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-1) AuditLogType: USER_CLEAR_ALL_DISMISSED_AUDIT_LOG not exist in string table >2014-07-22 17:03:59,519 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-1) AuditLogType: USER_CLEAR_ALL_DISMISSED_AUDIT_LOG_FAILED not exist in string table >2014-07-22 17:03:59,798 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-1) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-07-22 17:03:59,844 INFO [org.ovirt.engine.core.bll.LogoutBySessionCommand] (ajp--127.0.0.1-8702-1) [429a523e] Running command: LogoutBySessionCommand internal: false. >2014-07-22 17:03:59,864 INFO [org.ovirt.engine.core.bll.LogoutUserCommand] (ajp--127.0.0.1-8702-1) [2a696afa] Running command: LogoutUserCommand internal: false. >2014-07-22 17:03:59,883 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-1) [2a696afa] Correlation ID: 2a696afa, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-07-22 17:08:18,598 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp--127.0.0.1-8702-2) Running command: LoginUserCommand internal: false. >2014-07-22 17:08:18,609 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-2) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-07-22 17:08:18,625 INFO [org.ovirt.engine.core.bll.LogoutBySessionCommand] (ajp--127.0.0.1-8702-2) [6daa0b72] Running command: LogoutBySessionCommand internal: false. >2014-07-22 17:08:18,637 INFO [org.ovirt.engine.core.bll.LogoutUserCommand] (ajp--127.0.0.1-8702-2) [1c8ad795] Running command: LogoutUserCommand internal: false. >2014-07-22 17:08:18,651 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-2) [1c8ad795] Correlation ID: 1c8ad795, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-07-22 17:11:27,720 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp--127.0.0.1-8702-3) Running command: LoginUserCommand internal: false. >2014-07-22 17:11:27,731 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-3) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-07-22 17:11:27,741 INFO [org.ovirt.engine.core.bll.LogoutBySessionCommand] (ajp--127.0.0.1-8702-3) [56e5ba51] Running command: LogoutBySessionCommand internal: false. >2014-07-22 17:11:27,747 INFO [org.ovirt.engine.core.bll.LogoutUserCommand] (ajp--127.0.0.1-8702-3) [4cd9ad06] Running command: LogoutUserCommand internal: false. >2014-07-22 17:11:27,753 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-3) [4cd9ad06] Correlation ID: 4cd9ad06, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-07-22 17:14:17,831 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp--127.0.0.1-8702-4) Running command: LoginUserCommand internal: false. >2014-07-22 17:14:17,836 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-4) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-07-22 17:14:17,849 INFO [org.ovirt.engine.core.bll.LogoutBySessionCommand] (ajp--127.0.0.1-8702-4) [66ac6b48] Running command: LogoutBySessionCommand internal: false. >2014-07-22 17:14:17,861 INFO [org.ovirt.engine.core.bll.LogoutUserCommand] (ajp--127.0.0.1-8702-4) [51e93529] Running command: LogoutUserCommand internal: false. >2014-07-22 17:14:17,868 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-4) [51e93529] Correlation ID: 51e93529, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-07-22 18:03:28,330 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-51) The user admin from authz extension internal got updated since last interval >2014-07-22 18:03:28,423 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-57) Attempting to update VMs/Templates Ovf. >2014-07-22 19:03:28,340 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-18) The user admin from authz extension internal got updated since last interval >2014-07-22 19:03:28,431 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-11) Attempting to update VMs/Templates Ovf. >2014-07-22 20:03:28,348 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-73) The user admin from authz extension internal got updated since last interval >2014-07-22 20:03:28,434 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-77) Attempting to update VMs/Templates Ovf. >2014-07-22 21:03:28,355 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-30) The user admin from authz extension internal got updated since last interval >2014-07-22 21:03:28,437 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-23) Attempting to update VMs/Templates Ovf. >2014-07-22 22:03:28,365 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-98) The user admin from authz extension internal got updated since last interval >2014-07-22 22:03:28,442 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-90) Attempting to update VMs/Templates Ovf. >2014-07-22 23:03:28,375 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-38) The user admin from authz extension internal got updated since last interval >2014-07-22 23:03:28,445 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-49) Attempting to update VMs/Templates Ovf. >2014-07-23 00:03:28,382 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-11) The user admin from authz extension internal got updated since last interval >2014-07-23 00:03:28,448 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-12) Attempting to update VMs/Templates Ovf. >2014-07-23 01:03:28,389 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-86) The user admin from authz extension internal got updated since last interval >2014-07-23 01:03:28,449 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-64) Attempting to update VMs/Templates Ovf. >2014-07-23 02:03:28,395 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-31) The user admin from authz extension internal got updated since last interval >2014-07-23 02:03:28,452 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-40) Attempting to update VMs/Templates Ovf. >2014-07-23 03:03:28,402 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-91) The user admin from authz extension internal got updated since last interval >2014-07-23 03:03:28,455 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-82) Attempting to update VMs/Templates Ovf. >2014-07-23 03:35:35,002 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-18) Start deleteAgedOutCommandEntities >2014-07-23 03:35:35,001 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-1) Start deleteAgedOutAuditLogs >2014-07-23 03:35:35,015 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-18) Finished deleteAgedOutCommandEntities >2014-07-23 03:35:35,019 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-1) Finished deleteAgedOutAuditLogs >2014-07-23 04:03:28,411 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-53) The user admin from authz extension internal got updated since last interval >2014-07-23 04:03:28,456 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-51) Attempting to update VMs/Templates Ovf. >2014-07-23 05:03:28,417 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-39) The user admin from authz extension internal got updated since last interval >2014-07-23 05:03:28,458 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-16) Attempting to update VMs/Templates Ovf. >2014-07-23 06:03:28,422 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-68) The user admin from authz extension internal got updated since last interval >2014-07-23 06:03:28,460 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-65) Attempting to update VMs/Templates Ovf. >2014-07-23 07:03:28,428 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-1) The user admin from authz extension internal got updated since last interval >2014-07-23 07:03:28,463 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-18) Attempting to update VMs/Templates Ovf. >2014-07-23 08:03:28,437 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-84) The user admin from authz extension internal got updated since last interval >2014-07-23 08:03:28,464 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-35) Attempting to update VMs/Templates Ovf. >2014-07-23 09:03:28,444 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-61) The user admin from authz extension internal got updated since last interval >2014-07-23 09:03:28,468 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-54) Attempting to update VMs/Templates Ovf. >2014-07-23 10:03:28,455 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-2) The user admin from authz extension internal got updated since last interval >2014-07-23 10:03:28,471 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-16) Attempting to update VMs/Templates Ovf. >2014-07-23 11:03:28,466 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-46) The user admin from authz extension internal got updated since last interval >2014-07-23 11:03:28,473 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-79) Attempting to update VMs/Templates Ovf. >2014-07-23 12:03:28,475 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-99) Attempting to update VMs/Templates Ovf. >2014-07-23 12:03:28,481 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-31) The user admin from authz extension internal got updated since last interval >2014-07-23 13:03:28,477 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-4) Attempting to update VMs/Templates Ovf. >2014-07-23 13:03:28,491 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-92) The user admin from authz extension internal got updated since last interval >2014-07-23 14:03:28,481 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-95) Attempting to update VMs/Templates Ovf. >2014-07-23 14:03:28,497 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-85) The user admin from authz extension internal got updated since last interval >2014-07-23 15:03:28,483 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-42) Attempting to update VMs/Templates Ovf. >2014-07-23 15:03:28,502 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-96) The user admin from authz extension internal got updated since last interval >2014-07-23 16:03:28,485 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-68) Attempting to update VMs/Templates Ovf. >2014-07-23 16:03:28,512 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-62) The user admin from authz extension internal got updated since last interval >2014-07-23 17:03:28,488 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-9) Attempting to update VMs/Templates Ovf. >2014-07-23 17:03:28,523 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-16) The user admin from authz extension internal got updated since last interval >2014-07-23 18:03:28,490 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-65) Attempting to update VMs/Templates Ovf. >2014-07-23 18:03:28,535 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-83) The user admin from authz extension internal got updated since last interval >2014-07-23 19:03:28,493 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-43) Attempting to update VMs/Templates Ovf. >2014-07-23 19:03:28,542 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-36) The user admin from authz extension internal got updated since last interval >2014-07-23 20:03:28,495 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-1) Attempting to update VMs/Templates Ovf. >2014-07-23 20:03:28,549 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-40) The user admin from authz extension internal got updated since last interval >2014-07-23 21:03:28,498 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-35) Attempting to update VMs/Templates Ovf. >2014-07-23 21:03:28,559 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-60) The user admin from authz extension internal got updated since last interval >2014-07-23 22:03:28,500 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-24) Attempting to update VMs/Templates Ovf. >2014-07-23 22:03:28,568 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-57) The user admin from authz extension internal got updated since last interval >2014-07-23 23:03:28,503 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-59) Attempting to update VMs/Templates Ovf. >2014-07-23 23:03:28,574 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-98) The user admin from authz extension internal got updated since last interval >2014-07-24 00:03:28,506 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-58) Attempting to update VMs/Templates Ovf. >2014-07-24 00:03:28,583 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-67) The user admin from authz extension internal got updated since last interval >2014-07-24 01:03:28,508 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-82) Attempting to update VMs/Templates Ovf. >2014-07-24 01:03:28,589 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-11) The user admin from authz extension internal got updated since last interval >2014-07-24 02:03:28,512 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-60) Attempting to update VMs/Templates Ovf. >2014-07-24 02:03:28,594 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-7) The user admin from authz extension internal got updated since last interval >2014-07-24 03:03:28,515 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-23) Attempting to update VMs/Templates Ovf. >2014-07-24 03:03:28,601 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-6) The user admin from authz extension internal got updated since last interval >2014-07-24 03:35:35,001 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-26) Start deleteAgedOutAuditLogs >2014-07-24 03:35:35,001 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-60) Start deleteAgedOutCommandEntities >2014-07-24 03:35:35,007 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-26) Finished deleteAgedOutAuditLogs >2014-07-24 03:35:35,010 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-60) Finished deleteAgedOutCommandEntities >2014-07-24 04:03:28,517 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-73) Attempting to update VMs/Templates Ovf. >2014-07-24 04:03:28,608 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-59) The user admin from authz extension internal got updated since last interval >2014-07-24 05:03:28,520 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-62) Attempting to update VMs/Templates Ovf. >2014-07-24 05:03:28,613 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-64) The user admin from authz extension internal got updated since last interval >2014-07-24 06:03:28,522 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-66) Attempting to update VMs/Templates Ovf. >2014-07-24 06:03:28,618 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-96) The user admin from authz extension internal got updated since last interval >2014-07-24 07:03:28,524 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-28) Attempting to update VMs/Templates Ovf. >2014-07-24 07:03:28,626 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-53) The user admin from authz extension internal got updated since last interval >2014-07-24 08:03:28,527 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-32) Attempting to update VMs/Templates Ovf. >2014-07-24 08:03:28,633 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-39) The user admin from authz extension internal got updated since last interval >2014-07-24 09:03:28,529 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-100) Attempting to update VMs/Templates Ovf. >2014-07-24 09:03:28,639 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-7) The user admin from authz extension internal got updated since last interval >2014-07-24 09:34:22,185 INFO [org.ovirt.engine.core.bll.LoginAdminUserCommand] (ajp--127.0.0.1-8702-4) Running command: LoginAdminUserCommand internal: false. >2014-07-24 09:34:22,202 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-4) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-07-24 09:34:24,208 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp--127.0.0.1-8702-2) Running command: LoginUserCommand internal: false. >2014-07-24 09:40:48,018 INFO [org.ovirt.engine.core.bll.AddVdsCommand] (ajp--127.0.0.1-8702-5) [1270305f] Running command: AddVdsCommand internal: false. Entities affected : ID: 00000001-0001-0001-0001-0000000002c2 Type: VdsGroups >2014-07-24 09:40:48,147 INFO [org.ovirt.engine.core.bll.AddVdsSpmIdCommand] (ajp--127.0.0.1-8702-5) [680a0cb6] Running command: AddVdsSpmIdCommand internal: true. Entities affected : ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS >2014-07-24 09:40:48,166 INFO [org.ovirt.engine.core.bll.AddVdsSpmIdCommand] (ajp--127.0.0.1-8702-5) [680a0cb6] Lock freed to object EngineLock [exclusiveLocks= key: 00000002-0002-0002-0002-0000000001d9 value: REGISTER_VDS >, sharedLocks= ] >2014-07-24 09:40:48,175 INFO [org.ovirt.engine.core.vdsbroker.RemoveVdsVDSCommand] (ajp--127.0.0.1-8702-5) [680a0cb6] START, RemoveVdsVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 25de2f7b >2014-07-24 09:40:48,177 INFO [org.ovirt.engine.core.vdsbroker.RemoveVdsVDSCommand] (ajp--127.0.0.1-8702-5) [680a0cb6] FINISH, RemoveVdsVDSCommand, log id: 25de2f7b >2014-07-24 09:40:48,180 INFO [org.ovirt.engine.core.vdsbroker.AddVdsVDSCommand] (ajp--127.0.0.1-8702-5) [680a0cb6] START, AddVdsVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 75f74996 >2014-07-24 09:40:48,182 INFO [org.ovirt.engine.core.vdsbroker.AddVdsVDSCommand] (ajp--127.0.0.1-8702-5) [680a0cb6] AddVds - entered , starting logic to add VDS 17f7fe56-5f32-40ef-b098-611b84484e0b >2014-07-24 09:40:48,185 INFO [org.ovirt.engine.core.vdsbroker.AddVdsVDSCommand] (ajp--127.0.0.1-8702-5) [680a0cb6] AddVds - VDS 17f7fe56-5f32-40ef-b098-611b84484e0b was added, will try to add it to the resource manager >2014-07-24 09:40:48,191 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (ajp--127.0.0.1-8702-5) [680a0cb6] Entered VdsManager constructor >2014-07-24 09:40:48,214 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (ajp--127.0.0.1-8702-5) [680a0cb6] Initialize vdsBroker (10.34.62.208,54,321) >2014-07-24 09:40:48,368 INFO [org.ovirt.engine.core.vdsbroker.ResourceManager] (ajp--127.0.0.1-8702-5) [680a0cb6] VDS 17f7fe56-5f32-40ef-b098-611b84484e0b was added to the Resource Manager >2014-07-24 09:40:48,377 INFO [org.ovirt.engine.core.vdsbroker.AddVdsVDSCommand] (ajp--127.0.0.1-8702-5) [680a0cb6] FINISH, AddVdsVDSCommand, log id: 75f74996 >2014-07-24 09:40:48,394 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-5) [680a0cb6] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host rack. >2014-07-24 09:40:48,448 INFO [org.ovirt.engine.core.bll.InstallVdsInternalCommand] (org.ovirt.thread.pool-8-thread-8) [43a66d5e] Lock Acquired to object EngineLock [exclusiveLocks= key: 17f7fe56-5f32-40ef-b098-611b84484e0b value: VDS >, sharedLocks= ] >2014-07-24 09:40:48,453 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-5) [680a0cb6] Correlation ID: 1270305f, Job ID: 06b707e0-e985-4e05-a58a-ddecf62d1c41, Call Stack: null, Custom Event ID: -1, Message: Host rack was added by admin. >2014-07-24 09:40:48,475 INFO [org.ovirt.engine.core.bll.InstallVdsInternalCommand] (org.ovirt.thread.pool-8-thread-8) [43a66d5e] Running command: InstallVdsInternalCommand internal: true. Entities affected : ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS >2014-07-24 09:40:48,488 INFO [org.ovirt.engine.core.bll.InstallVdsInternalCommand] (org.ovirt.thread.pool-8-thread-8) [43a66d5e] Before Installation host 17f7fe56-5f32-40ef-b098-611b84484e0b, rack >2014-07-24 09:40:48,504 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-8-thread-8) [43a66d5e] START, SetVdsStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, status=Installing, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 1388ca8d >2014-07-24 09:40:48,533 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-8-thread-8) [43a66d5e] FINISH, SetVdsStatusVDSCommand, log id: 1388ca8d >2014-07-24 09:40:48,595 INFO [org.ovirt.engine.core.bll.InstallerMessages] (org.ovirt.thread.pool-8-thread-8) [43a66d5e] Installation 10.34.62.208: Connected to host 10.34.62.208 with SSH key fingerprint: 3f:fe:39:f2:21:7b:9c:a0:53:49:a6:35:b0:6c:4a:ff >2014-07-24 09:40:48,652 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-8) [43a66d5e] Correlation ID: 43a66d5e, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Connected to host 10.34.62.208 with SSH key fingerprint: 3f:fe:39:f2:21:7b:9c:a0:53:49:a6:35:b0:6c:4a:ff. >2014-07-24 09:40:48,672 INFO [org.ovirt.engine.core.bll.VdsDeploy] (org.ovirt.thread.pool-8-thread-8) [43a66d5e] Installation of 10.34.62.208. Executing command via SSH umask 0077; MYTMP="$(TMPDIR="${OVIRT_TMPDIR}" mktemp -t ovirt-XXXXXXXXXX)"; trap "chmod -R u+rwX \"${MYTMP}\" > /dev/null 2>&1; rm -fr \"${MYTMP}\" > /dev/null 2>&1" 0; rm -fr "${MYTMP}" && mkdir "${MYTMP}" && tar --warning=no-timestamp -C "${MYTMP}" -x && "${MYTMP}"/setup DIALOG/dialect=str:machine DIALOG/customization=bool:True < /var/cache/ovirt-engine/ovirt-host-deploy.tar >2014-07-24 09:40:48,676 INFO [org.ovirt.engine.core.utils.archivers.tar.CachedTar] (org.ovirt.thread.pool-8-thread-8) Tarball '/var/cache/ovirt-engine/ovirt-host-deploy.tar' is missing, creating >2014-07-24 09:40:48,807 INFO [org.ovirt.engine.core.uutils.ssh.SSHDialog] (org.ovirt.thread.pool-8-thread-8) SSH execute root@10.34.62.208 'umask 0077; MYTMP="$(TMPDIR="${OVIRT_TMPDIR}" mktemp -t ovirt-XXXXXXXXXX)"; trap "chmod -R u+rwX \"${MYTMP}\" > /dev/null 2>&1; rm -fr \"${MYTMP}\" > /dev/null 2>&1" 0; rm -fr "${MYTMP}" && mkdir "${MYTMP}" && tar --warning=no-timestamp -C "${MYTMP}" -x && "${MYTMP}"/setup DIALOG/dialect=str:machine DIALOG/customization=bool:True' >2014-07-24 09:40:49,915 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.34.62.208: Stage: Initializing >2014-07-24 09:40:49,923 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 43a66d5e, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Stage: Initializing. >2014-07-24 09:40:49,937 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.34.62.208: Stage: Environment setup >2014-07-24 09:40:49,945 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 43a66d5e, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Stage: Environment setup. >2014-07-24 09:40:49,955 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.34.62.208: Stage: Environment packages setup >2014-07-24 09:40:49,962 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 43a66d5e, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Stage: Environment packages setup. >2014-07-24 09:40:51,371 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-6) Initializing Host: rack >2014-07-24 09:41:24,336 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.34.62.208: Yum Downloading: ovirt-3.5-epel/metalink (0%) >2014-07-24 09:41:24,344 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 43a66d5e, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Downloading: ovirt-3.5-epel/metalink (0%). >2014-07-24 09:41:32,025 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.34.62.208: Stage: Programs detection >2014-07-24 09:41:32,032 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 43a66d5e, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Stage: Programs detection. >2014-07-24 09:41:32,033 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.34.62.208: Stage: Environment customization >2014-07-24 09:41:32,038 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 43a66d5e, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Stage: Environment customization. >2014-07-24 09:42:02,467 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.34.62.208: Kdump supported >2014-07-24 09:42:02,475 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 43a66d5e, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Kdump supported. >2014-07-24 09:42:02,498 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.34.62.208: Logs at host located at: '/tmp/ovirt-host-deploy-20140724094049-nekf10.log' >2014-07-24 09:42:02,505 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 43a66d5e, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Logs at host located at: '/tmp/ovirt-host-deploy-20140724094049-nekf10.log'. >2014-07-24 09:42:02,598 INFO [org.ovirt.engine.core.bll.VdsDeploy] (VdsDeploy) Host 10.34.62.208 reports unique id 4C4C4544-0034-5310-8052-B4C04F4A354A >2014-07-24 09:42:02,629 INFO [org.ovirt.engine.core.bll.VdsDeploy] (VdsDeploy) Assigning unique id 4C4C4544-0034-5310-8052-B4C04F4A354A to Host 10.34.62.208 >2014-07-24 09:42:02,713 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.34.62.208: Stage: Setup validation >2014-07-24 09:42:02,719 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 43a66d5e, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Stage: Setup validation. >2014-07-24 09:42:02,755 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.34.62.208: Hardware supports virtualization >2014-07-24 09:42:02,761 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 43a66d5e, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Hardware supports virtualization. >2014-07-24 09:42:03,007 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.34.62.208: Stage: Transaction setup >2014-07-24 09:42:03,016 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 43a66d5e, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Stage: Transaction setup. >2014-07-24 09:42:03,061 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.34.62.208: Stage: Misc configuration >2014-07-24 09:42:03,070 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 43a66d5e, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Stage: Misc configuration. >2014-07-24 09:42:03,071 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.34.62.208: Stage: Package installation >2014-07-24 09:42:03,078 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 43a66d5e, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Stage: Package installation. >2014-07-24 09:42:44,942 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.34.62.208: Yum Status: Downloading Packages >2014-07-24 09:42:44,952 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 43a66d5e, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Status: Downloading Packages. >2014-07-24 09:42:44,968 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.34.62.208: Yum Download/Verify: ioprocess-0.5.0-1.el6.x86_64 >2014-07-24 09:42:44,975 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 43a66d5e, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Download/Verify: ioprocess-0.5.0-1.el6.x86_64. >2014-07-24 09:42:46,340 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.34.62.208: Yum Download/Verify: kexec-tools-2.0.0-273.1.el6.x86_64 >2014-07-24 09:42:46,347 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 43a66d5e, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Download/Verify: kexec-tools-2.0.0-273.1.el6.x86_64. >2014-07-24 09:42:46,349 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.34.62.208: Yum Download/Verify: mom-0.4.1-2.el6.noarch >2014-07-24 09:42:46,355 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 43a66d5e, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Download/Verify: mom-0.4.1-2.el6.noarch. >2014-07-24 09:42:46,366 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.34.62.208: Yum Download/Verify: python-ioprocess-0.5.0-1.el6.noarch >2014-07-24 09:42:46,373 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 43a66d5e, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Download/Verify: python-ioprocess-0.5.0-1.el6.noarch. >2014-07-24 09:42:47,996 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.34.62.208: Yum Download/Verify: 2:qemu-kvm-rhev-tools-0.12.1.2-2.415.el6_5.12.x86_64 >2014-07-24 09:42:48,004 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 43a66d5e, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Download/Verify: 2:qemu-kvm-rhev-tools-0.12.1.2-2.415.el6_5.12.x86_64. >2014-07-24 09:42:49,776 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.34.62.208: Yum Download/Verify: vdsm-4.16.0-42.git3bfad86.el6.x86_64 >2014-07-24 09:42:49,787 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 43a66d5e, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Download/Verify: vdsm-4.16.0-42.git3bfad86.el6.x86_64. >2014-07-24 09:42:49,989 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.34.62.208: Yum Download/Verify: vdsm-cli-4.16.0-42.git3bfad86.el6.noarch >2014-07-24 09:42:49,998 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 43a66d5e, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Download/Verify: vdsm-cli-4.16.0-42.git3bfad86.el6.noarch. >2014-07-24 09:42:50,408 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.34.62.208: Yum Download/Verify: vdsm-jsonrpc-4.16.0-42.git3bfad86.el6.noarch >2014-07-24 09:42:50,417 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 43a66d5e, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Download/Verify: vdsm-jsonrpc-4.16.0-42.git3bfad86.el6.noarch. >2014-07-24 09:42:50,789 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.34.62.208: Yum Download/Verify: vdsm-python-4.16.0-42.git3bfad86.el6.x86_64 >2014-07-24 09:42:50,796 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 43a66d5e, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Download/Verify: vdsm-python-4.16.0-42.git3bfad86.el6.x86_64. >2014-07-24 09:42:50,977 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.34.62.208: Yum Download/Verify: vdsm-python-zombiereaper-4.16.0-42.git3bfad86.el6.noarch >2014-07-24 09:42:50,984 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 43a66d5e, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Download/Verify: vdsm-python-zombiereaper-4.16.0-42.git3bfad86.el6.noarch. >2014-07-24 09:42:51,197 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.34.62.208: Yum Download/Verify: vdsm-xmlrpc-4.16.0-42.git3bfad86.el6.noarch >2014-07-24 09:42:51,210 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 43a66d5e, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Download/Verify: vdsm-xmlrpc-4.16.0-42.git3bfad86.el6.noarch. >2014-07-24 09:42:51,381 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.34.62.208: Yum Download/Verify: vdsm-yajsonrpc-4.16.0-42.git3bfad86.el6.noarch >2014-07-24 09:42:51,387 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 43a66d5e, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Download/Verify: vdsm-yajsonrpc-4.16.0-42.git3bfad86.el6.noarch. >2014-07-24 09:42:51,506 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.34.62.208: Yum Status: Check Package Signatures >2014-07-24 09:42:51,515 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 43a66d5e, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Status: Check Package Signatures. >2014-07-24 09:42:51,523 WARN [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.34.62.208: Confirm use of GPG Key userid=oVirt <infra@ovirt.org> hexkeyid=FE590CB7 >2014-07-24 09:42:51,532 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 43a66d5e, Call Stack: null, Custom Event ID: -1, Message: Host rack installation in progress . Confirm use of GPG Key userid=oVirt <infra@ovirt.org> hexkeyid=FE590CB7. >2014-07-24 09:42:51,664 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.34.62.208: Yum Status: Running Test Transaction >2014-07-24 09:42:51,673 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 43a66d5e, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Status: Running Test Transaction. >2014-07-24 09:42:51,963 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.34.62.208: Yum Status: Running Transaction >2014-07-24 09:42:51,970 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 43a66d5e, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Status: Running Transaction. >2014-07-24 09:42:52,404 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.34.62.208: Yum update: 1/22: vdsm-python-zombiereaper-4.16.0-42.git3bfad86.el6.noarch >2014-07-24 09:42:52,410 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 43a66d5e, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum update: 1/22: vdsm-python-zombiereaper-4.16.0-42.git3bfad86.el6.noarch. >2014-07-24 09:42:52,691 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.34.62.208: Yum install: 2/22: ioprocess-0.5.0-1.el6.x86_64 >2014-07-24 09:42:52,698 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 43a66d5e, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum install: 2/22: ioprocess-0.5.0-1.el6.x86_64. >2014-07-24 09:42:53,068 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.34.62.208: Yum install: 3/22: python-ioprocess-0.5.0-1.el6.noarch >2014-07-24 09:42:53,078 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 43a66d5e, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum install: 3/22: python-ioprocess-0.5.0-1.el6.noarch. >2014-07-24 09:42:53,390 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.34.62.208: Yum update: 4/22: vdsm-python-4.16.0-42.git3bfad86.el6.x86_64 >2014-07-24 09:42:53,397 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 43a66d5e, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum update: 4/22: vdsm-python-4.16.0-42.git3bfad86.el6.x86_64. >2014-07-24 09:42:53,907 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.34.62.208: Yum update: 5/22: vdsm-xmlrpc-4.16.0-42.git3bfad86.el6.noarch >2014-07-24 09:42:53,913 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 43a66d5e, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum update: 5/22: vdsm-xmlrpc-4.16.0-42.git3bfad86.el6.noarch. >2014-07-24 09:42:54,207 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.34.62.208: Yum update: 6/22: mom-0.4.1-2.el6.noarch >2014-07-24 09:42:54,212 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 43a66d5e, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum update: 6/22: mom-0.4.1-2.el6.noarch. >2014-07-24 09:42:54,799 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.34.62.208: Yum update: 7/22: vdsm-yajsonrpc-4.16.0-42.git3bfad86.el6.noarch >2014-07-24 09:42:54,807 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 43a66d5e, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum update: 7/22: vdsm-yajsonrpc-4.16.0-42.git3bfad86.el6.noarch. >2014-07-24 09:42:55,074 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.34.62.208: Yum update: 8/22: vdsm-jsonrpc-4.16.0-42.git3bfad86.el6.noarch >2014-07-24 09:42:55,080 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 43a66d5e, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum update: 8/22: vdsm-jsonrpc-4.16.0-42.git3bfad86.el6.noarch. >2014-07-24 09:42:55,514 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.34.62.208: Yum update: 9/22: vdsm-4.16.0-42.git3bfad86.el6.x86_64 >2014-07-24 09:42:55,521 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 43a66d5e, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum update: 9/22: vdsm-4.16.0-42.git3bfad86.el6.x86_64. >2014-07-24 09:43:04,446 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.34.62.208: Yum update: 10/22: vdsm-cli-4.16.0-42.git3bfad86.el6.noarch >2014-07-24 09:43:04,452 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 43a66d5e, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum update: 10/22: vdsm-cli-4.16.0-42.git3bfad86.el6.noarch. >2014-07-24 09:43:04,791 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.34.62.208: Yum update: 11/22: 2:qemu-kvm-rhev-tools-0.12.1.2-2.415.el6_5.12.x86_64 >2014-07-24 09:43:04,798 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 43a66d5e, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum update: 11/22: 2:qemu-kvm-rhev-tools-0.12.1.2-2.415.el6_5.12.x86_64. >2014-07-24 09:43:05,135 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.34.62.208: Yum update: 12/22: kexec-tools-2.0.0-273.1.el6.x86_64 >2014-07-24 09:43:05,143 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 43a66d5e, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum update: 12/22: kexec-tools-2.0.0-273.1.el6.x86_64. >2014-07-24 09:43:19,524 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.34.62.208: Yum updated: 13/22: vdsm-cli >2014-07-24 09:43:19,532 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 43a66d5e, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum updated: 13/22: vdsm-cli. >2014-07-24 09:43:19,831 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.34.62.208: Yum updated: 14/22: vdsm >2014-07-24 09:43:19,837 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 43a66d5e, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum updated: 14/22: vdsm. >2014-07-24 09:43:20,985 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.34.62.208: Yum updated: 15/22: vdsm-jsonrpc >2014-07-24 09:43:20,993 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 43a66d5e, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum updated: 15/22: vdsm-jsonrpc. >2014-07-24 09:43:21,304 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.34.62.208: Yum updated: 16/22: vdsm-xmlrpc >2014-07-24 09:43:21,310 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 43a66d5e, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum updated: 16/22: vdsm-xmlrpc. >2014-07-24 09:43:21,492 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.34.62.208: Yum updated: 17/22: vdsm-python >2014-07-24 09:43:21,499 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 43a66d5e, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum updated: 17/22: vdsm-python. >2014-07-24 09:43:21,820 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.34.62.208: Yum updated: 18/22: vdsm-python-zombiereaper >2014-07-24 09:43:21,838 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 43a66d5e, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum updated: 18/22: vdsm-python-zombiereaper. >2014-07-24 09:43:22,015 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.34.62.208: Yum updated: 19/22: vdsm-yajsonrpc >2014-07-24 09:43:22,022 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 43a66d5e, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum updated: 19/22: vdsm-yajsonrpc. >2014-07-24 09:43:22,236 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.34.62.208: Yum updated: 20/22: mom >2014-07-24 09:43:22,242 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 43a66d5e, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum updated: 20/22: mom. >2014-07-24 09:43:22,616 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.34.62.208: Yum updated: 21/22: qemu-kvm-rhev-tools >2014-07-24 09:43:22,622 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 43a66d5e, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum updated: 21/22: qemu-kvm-rhev-tools. >2014-07-24 09:43:22,858 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.34.62.208: Yum updated: 22/22: kexec-tools >2014-07-24 09:43:22,864 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 43a66d5e, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum updated: 22/22: kexec-tools. >2014-07-24 09:43:27,395 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.34.62.208: Yum Verify: 1/22: vdsm-python-zombiereaper.noarch 0:4.16.0-42.git3bfad86.el6 - u >2014-07-24 09:43:27,402 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 43a66d5e, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Verify: 1/22: vdsm-python-zombiereaper.noarch 0:4.16.0-42.git3bfad86.el6 - u. >2014-07-24 09:43:27,497 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.34.62.208: Yum Verify: 2/22: python-ioprocess.noarch 0:0.5.0-1.el6 - u >2014-07-24 09:43:27,505 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 43a66d5e, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Verify: 2/22: python-ioprocess.noarch 0:0.5.0-1.el6 - u. >2014-07-24 09:43:27,655 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.34.62.208: Yum Verify: 3/22: vdsm-python.x86_64 0:4.16.0-42.git3bfad86.el6 - u >2014-07-24 09:43:27,663 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 43a66d5e, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Verify: 3/22: vdsm-python.x86_64 0:4.16.0-42.git3bfad86.el6 - u. >2014-07-24 09:43:27,665 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.34.62.208: Yum Verify: 4/22: vdsm-cli.noarch 0:4.16.0-42.git3bfad86.el6 - u >2014-07-24 09:43:27,672 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 43a66d5e, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Verify: 4/22: vdsm-cli.noarch 0:4.16.0-42.git3bfad86.el6 - u. >2014-07-24 09:43:27,716 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.34.62.208: Yum Verify: 5/22: vdsm.x86_64 0:4.16.0-42.git3bfad86.el6 - u >2014-07-24 09:43:27,722 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 43a66d5e, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Verify: 5/22: vdsm.x86_64 0:4.16.0-42.git3bfad86.el6 - u. >2014-07-24 09:43:27,816 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.34.62.208: Yum Verify: 6/22: kexec-tools.x86_64 0:2.0.0-273.1.el6 - u >2014-07-24 09:43:27,822 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 43a66d5e, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Verify: 6/22: kexec-tools.x86_64 0:2.0.0-273.1.el6 - u. >2014-07-24 09:43:27,901 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.34.62.208: Yum Verify: 7/22: vdsm-jsonrpc.noarch 0:4.16.0-42.git3bfad86.el6 - u >2014-07-24 09:43:27,909 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 43a66d5e, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Verify: 7/22: vdsm-jsonrpc.noarch 0:4.16.0-42.git3bfad86.el6 - u. >2014-07-24 09:43:27,970 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.34.62.208: Yum Verify: 8/22: qemu-kvm-rhev-tools.x86_64 2:0.12.1.2-2.415.el6_5.12 - u >2014-07-24 09:43:27,977 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 43a66d5e, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Verify: 8/22: qemu-kvm-rhev-tools.x86_64 2:0.12.1.2-2.415.el6_5.12 - u. >2014-07-24 09:43:28,044 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.34.62.208: Yum Verify: 9/22: vdsm-yajsonrpc.noarch 0:4.16.0-42.git3bfad86.el6 - u >2014-07-24 09:43:28,052 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 43a66d5e, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Verify: 9/22: vdsm-yajsonrpc.noarch 0:4.16.0-42.git3bfad86.el6 - u. >2014-07-24 09:43:28,111 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.34.62.208: Yum Verify: 10/22: vdsm-xmlrpc.noarch 0:4.16.0-42.git3bfad86.el6 - u >2014-07-24 09:43:28,119 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 43a66d5e, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Verify: 10/22: vdsm-xmlrpc.noarch 0:4.16.0-42.git3bfad86.el6 - u. >2014-07-24 09:43:28,179 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.34.62.208: Yum Verify: 11/22: mom.noarch 0:0.4.1-2.el6 - u >2014-07-24 09:43:28,188 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 43a66d5e, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Verify: 11/22: mom.noarch 0:0.4.1-2.el6 - u. >2014-07-24 09:43:28,256 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.34.62.208: Yum Verify: 12/22: ioprocess.x86_64 0:0.5.0-1.el6 - u >2014-07-24 09:43:28,264 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 43a66d5e, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Verify: 12/22: ioprocess.x86_64 0:0.5.0-1.el6 - u. >2014-07-24 09:43:28,304 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.34.62.208: Yum Verify: 13/22: vdsm-yajsonrpc.noarch 0:4.15.0-106.git78c8b21.el6 - ud >2014-07-24 09:43:28,313 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 43a66d5e, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Verify: 13/22: vdsm-yajsonrpc.noarch 0:4.15.0-106.git78c8b21.el6 - ud. >2014-07-24 09:43:28,315 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.34.62.208: Yum Verify: 14/22: kexec-tools.x86_64 0:2.0.0-273.el6 - ud >2014-07-24 09:43:28,321 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 43a66d5e, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Verify: 14/22: kexec-tools.x86_64 0:2.0.0-273.el6 - ud. >2014-07-24 09:43:28,322 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.34.62.208: Yum Verify: 15/22: mom.noarch 0:0.4.0-1.el6ev - ud >2014-07-24 09:43:28,328 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 43a66d5e, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Verify: 15/22: mom.noarch 0:0.4.0-1.el6ev - ud. >2014-07-24 09:43:28,329 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.34.62.208: Yum Verify: 16/22: vdsm-jsonrpc.noarch 0:4.15.0-106.git78c8b21.el6 - ud >2014-07-24 09:43:28,335 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 43a66d5e, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Verify: 16/22: vdsm-jsonrpc.noarch 0:4.15.0-106.git78c8b21.el6 - ud. >2014-07-24 09:43:28,337 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.34.62.208: Yum Verify: 17/22: qemu-kvm-rhev-tools.x86_64 2:0.12.1.2-2.415.el6_5.10 - ud >2014-07-24 09:43:28,343 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 43a66d5e, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Verify: 17/22: qemu-kvm-rhev-tools.x86_64 2:0.12.1.2-2.415.el6_5.10 - ud. >2014-07-24 09:43:28,344 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.34.62.208: Yum Verify: 18/22: vdsm-cli.noarch 0:4.15.0-106.git78c8b21.el6 - ud >2014-07-24 09:43:28,352 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 43a66d5e, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Verify: 18/22: vdsm-cli.noarch 0:4.15.0-106.git78c8b21.el6 - ud. >2014-07-24 09:43:28,353 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.34.62.208: Yum Verify: 19/22: vdsm-python.x86_64 0:4.15.0-106.git78c8b21.el6 - ud >2014-07-24 09:43:28,359 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 43a66d5e, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Verify: 19/22: vdsm-python.x86_64 0:4.15.0-106.git78c8b21.el6 - ud. >2014-07-24 09:43:28,360 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.34.62.208: Yum Verify: 20/22: vdsm.x86_64 0:4.15.0-106.git78c8b21.el6 - ud >2014-07-24 09:43:28,368 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 43a66d5e, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Verify: 20/22: vdsm.x86_64 0:4.15.0-106.git78c8b21.el6 - ud. >2014-07-24 09:43:28,370 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.34.62.208: Yum Verify: 21/22: vdsm-python-zombiereaper.noarch 0:4.15.0-106.git78c8b21.el6 - ud >2014-07-24 09:43:28,375 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 43a66d5e, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Verify: 21/22: vdsm-python-zombiereaper.noarch 0:4.15.0-106.git78c8b21.el6 - ud. >2014-07-24 09:43:28,377 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.34.62.208: Yum Verify: 22/22: vdsm-xmlrpc.noarch 0:4.15.0-106.git78c8b21.el6 - ud >2014-07-24 09:43:28,382 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 43a66d5e, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Yum Verify: 22/22: vdsm-xmlrpc.noarch 0:4.15.0-106.git78c8b21.el6 - ud. >2014-07-24 09:43:28,440 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.34.62.208: Stage: Misc configuration >2014-07-24 09:43:28,447 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 43a66d5e, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Stage: Misc configuration. >2014-07-24 09:43:28,530 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.34.62.208: Setting time >2014-07-24 09:43:28,536 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 43a66d5e, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Setting time. >2014-07-24 09:43:30,148 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.34.62.208: Enrolling certificate >2014-07-24 09:43:30,154 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 43a66d5e, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Enrolling certificate. >2014-07-24 09:43:31,498 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.34.62.208: Stage: Transaction commit >2014-07-24 09:43:31,505 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 43a66d5e, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Stage: Transaction commit. >2014-07-24 09:43:31,634 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.34.62.208: Stage: Closing up >2014-07-24 09:43:31,641 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 43a66d5e, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Stage: Closing up. >2014-07-24 09:43:32,051 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.34.62.208: Restarting kdump >2014-07-24 09:43:32,058 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 43a66d5e, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Restarting kdump. >2014-07-24 09:43:45,267 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.34.62.208: Stopping libvirtd >2014-07-24 09:43:45,273 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 43a66d5e, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Stopping libvirtd. >2014-07-24 09:43:45,534 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.34.62.208: Starting vdsm >2014-07-24 09:43:45,540 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 43a66d5e, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Starting vdsm. >2014-07-24 09:43:47,326 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.34.62.208: Stage: Pre-termination >2014-07-24 09:43:47,338 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 43a66d5e, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Stage: Pre-termination. >2014-07-24 09:43:47,349 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.34.62.208: Retrieving installation logs to: '/var/log/ovirt-engine/host-deploy/ovirt-20140724094347-10.34.62.208-43a66d5e.log' >2014-07-24 09:43:47,358 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 43a66d5e, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Retrieving installation logs to: '/var/log/ovirt-engine/host-deploy/ovirt-20140724094347-10.34.62.208-43a66d5e.log'. >2014-07-24 09:43:47,624 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.34.62.208: Stage: Termination >2014-07-24 09:43:47,631 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 43a66d5e, Call Stack: null, Custom Event ID: -1, Message: Installing Host rack. Stage: Termination. >2014-07-24 09:43:48,127 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.PollVDSCommand] (org.ovirt.thread.pool-8-thread-8) [43a66d5e] org.ovirt.engine.core.vdsbroker.vdsbroker.VDSRecoveringException: Recovering from crash or Initializing >2014-07-24 09:43:48,142 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.PollVDSCommand] (org.ovirt.thread.pool-8-thread-8) [43a66d5e] Command PollVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b) execution failed. Exception: VDSRecoveringException: Recovering from crash or Initializing >2014-07-24 09:43:48,669 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.PollVDSCommand] (org.ovirt.thread.pool-8-thread-8) [43a66d5e] org.ovirt.engine.core.vdsbroker.vdsbroker.VDSRecoveringException: Recovering from crash or Initializing >2014-07-24 09:43:48,673 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.PollVDSCommand] (org.ovirt.thread.pool-8-thread-8) [43a66d5e] Command PollVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b) execution failed. Exception: VDSRecoveringException: Recovering from crash or Initializing >2014-07-24 09:43:49,196 INFO [org.ovirt.engine.core.bll.network.NetworkConfigurator] (org.ovirt.thread.pool-8-thread-8) [43a66d5e] Engine managed to communicate with VDSM agent on host rack >2014-07-24 09:43:49,632 INFO [org.ovirt.engine.core.bll.network.NetworkConfigurator] (org.ovirt.thread.pool-8-thread-8) [43a66d5e] The management network ovirtmgmt is already configured on host rack >2014-07-24 09:43:49,635 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-8-thread-8) [43a66d5e] START, SetVdsStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, status=Initializing, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 596b9dcb >2014-07-24 09:43:49,642 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-8-thread-8) [43a66d5e] FINISH, SetVdsStatusVDSCommand, log id: 596b9dcb >2014-07-24 09:43:49,643 INFO [org.ovirt.engine.core.bll.InstallVdsInternalCommand] (org.ovirt.thread.pool-8-thread-8) [43a66d5e] After Installation host rack, VDS >2014-07-24 09:43:49,659 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-8) [43a66d5e] Correlation ID: 43a66d5e, Job ID: 06b707e0-e985-4e05-a58a-ddecf62d1c41, Call Stack: null, Custom Event ID: -1, Message: Host rack installed >2014-07-24 09:43:49,676 INFO [org.ovirt.engine.core.bll.InstallVdsInternalCommand] (org.ovirt.thread.pool-8-thread-8) [43a66d5e] Lock freed to object EngineLock [exclusiveLocks= key: 17f7fe56-5f32-40ef-b098-611b84484e0b value: VDS >, sharedLocks= ] >2014-07-24 09:43:51,819 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-50) START, GetHardwareInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]), log id: 30cf88c4 >2014-07-24 09:43:51,942 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-50) FINISH, GetHardwareInfoVDSCommand, log id: 30cf88c4 >2014-07-24 09:43:52,016 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-50) [6386d02e] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS >2014-07-24 09:43:52,141 INFO [org.ovirt.engine.core.bll.UpdateVdsGroupCommand] (DefaultQuartzScheduler_Worker-50) [6aa9f65f] Running command: UpdateVdsGroupCommand internal: true. Entities affected : ID: 00000001-0001-0001-0001-0000000002c2 Type: VdsGroups >2014-07-24 09:43:52,176 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-50) [6aa9f65f] Correlation ID: 6aa9f65f, Call Stack: null, Custom Event ID: -1, Message: Host cluster Default was updated by system >2014-07-24 09:43:52,221 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-50) [1c293098] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS >2014-07-24 09:43:52,225 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-50) [1c293098] START, SetVdsStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, status=NonOperational, nonOperationalReason=RNG_SOURCES_INCOMPATIBLE_WITH_CLUSTER, stopSpmFailureLogged=false), log id: 70eeacec >2014-07-24 09:43:52,232 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-50) [1c293098] FINISH, SetVdsStatusVDSCommand, log id: 70eeacec >2014-07-24 09:43:52,258 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-50) [1c293098] Correlation ID: 1c293098, Job ID: a3e8abcc-1329-43dd-b37b-ecbb43a04a3a, Call Stack: null, Custom Event ID: -1, Message: Host rack does not comply with the cluster Default Random Number Generator sources. The Hosts supported sources are: Non interactive user; and the cluster requirements are: RANDOM. >2014-07-24 09:43:52,324 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-50) [1c293098] START, GetHardwareInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]), log id: 2f58d973 >2014-07-24 09:43:52,372 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-50) [1c293098] FINISH, GetHardwareInfoVDSCommand, log id: 2f58d973 >2014-07-24 09:43:52,406 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-50) [11d4a56d] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS >2014-07-24 09:43:52,421 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-50) [11d4a56d] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host rack was set to NonOperational. >2014-07-24 09:43:52,462 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVdsCommand] (DefaultQuartzScheduler_Worker-50) [11d4a56d] START, FullListVdsCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b], vmIds=[a2660663-ef34-41ee-a580-0fc425183c36]), log id: 2472108a >2014-07-24 09:43:52,484 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVdsCommand] (DefaultQuartzScheduler_Worker-50) [11d4a56d] FINISH, FullListVdsCommand, return: [{displaySecurePort=-1, kvmEnable=true, nicModel=rtl8139,pv, guestIPs=, keyboardLayout=en-us, displayIp=10.34.62.208, pauseCode=NOERR, nice=0, displayNetwork=ovirtmgmt, username=, copyPasteEnable=true, timeOffset=7200, transparentHugePages=true, vmId=a2660663-ef34-41ee-a580-0fc425183c36, acpiEnable=true, custom={device_0a15b13b-427e-4147-9179-0dcdc4094307device_16e5639c-8110-45fa-b60d-ab8598d0ed4bdevice_3f51e57f-378c-4c48-9190-8893bc06a9a8device_43b633f0-4db0-4d8f-b872-fb2f6b5657a7device_62de13f5-3461-4945-9cec-d2cc141a05d0device_7ee4ae57-89f0-4b38-8f28-fc4a3ed37f7d=VmDevice {vmId=a2660663-ef34-41ee-a580-0fc425183c36, deviceId=7ee4ae57-89f0-4b38-8f28-fc4a3ed37f7d, device=virtio-serial, type=CONTROLLER, bootOrder=0, specParams={}, address={slot=0x05, bus=0x00, domain=0x0000, type=pci, function=0x0}, managed=false, plugged=true, readOnly=false, deviceAlias=virtio-serial0, customProperties={}, snapshotId=null}, device_0a15b13b-427e-4147-9179-0dcdc4094307device_16e5639c-8110-45fa-b60d-ab8598d0ed4bdevice_3f51e57f-378c-4c48-9190-8893bc06a9a8device_43b633f0-4db0-4d8f-b872-fb2f6b5657a7device_62de13f5-3461-4945-9cec-d2cc141a05d0=VmDevice {vmId=a2660663-ef34-41ee-a580-0fc425183c36, deviceId=62de13f5-3461-4945-9cec-d2cc141a05d0, device=unix, type=CHANNEL, bootOrder=0, specParams={}, address={bus=0, controller=0, type=virtio-serial, port=1}, managed=false, plugged=true, readOnly=false, deviceAlias=channel0, customProperties={}, snapshotId=null}, device_0a15b13b-427e-4147-9179-0dcdc4094307device_16e5639c-8110-45fa-b60d-ab8598d0ed4b=VmDevice {vmId=a2660663-ef34-41ee-a580-0fc425183c36, deviceId=16e5639c-8110-45fa-b60d-ab8598d0ed4b, device=spicevmc, type=CHANNEL, bootOrder=0, specParams={}, address={bus=0, controller=0, type=virtio-serial, port=3}, managed=false, plugged=true, readOnly=false, deviceAlias=channel2, customProperties={}, snapshotId=null}, device_0a15b13b-427e-4147-9179-0dcdc4094307device_16e5639c-8110-45fa-b60d-ab8598d0ed4bdevice_3f51e57f-378c-4c48-9190-8893bc06a9a8=VmDevice {vmId=a2660663-ef34-41ee-a580-0fc425183c36, deviceId=3f51e57f-378c-4c48-9190-8893bc06a9a8, device=ide, type=CONTROLLER, bootOrder=0, specParams={}, address={slot=0x01, bus=0x00, domain=0x0000, type=pci, function=0x1}, managed=false, plugged=true, readOnly=false, deviceAlias=ide0, customProperties={}, snapshotId=null}, device_0a15b13b-427e-4147-9179-0dcdc4094307device_16e5639c-8110-45fa-b60d-ab8598d0ed4bdevice_3f51e57f-378c-4c48-9190-8893bc06a9a8device_43b633f0-4db0-4d8f-b872-fb2f6b5657a7=VmDevice {vmId=a2660663-ef34-41ee-a580-0fc425183c36, deviceId=43b633f0-4db0-4d8f-b872-fb2f6b5657a7, device=unix, type=CHANNEL, bootOrder=0, specParams={}, address={bus=0, controller=0, type=virtio-serial, port=2}, managed=false, plugged=true, readOnly=false, deviceAlias=channel1, customProperties={}, snapshotId=null}, device_0a15b13b-427e-4147-9179-0dcdc4094307=VmDevice {vmId=a2660663-ef34-41ee-a580-0fc425183c36, deviceId=0a15b13b-427e-4147-9179-0dcdc4094307, device=memballoon, type=BALLOON, bootOrder=0, specParams={model=virtio}, address={slot=0x07, bus=0x00, domain=0x0000, type=pci, function=0x0}, managed=false, plugged=true, readOnly=false, deviceAlias=balloon0, customProperties={}, snapshotId=null}}, spiceSslCipherSuite=DEFAULT, memSize=5192, displayPort=-1, smp=1, status=Down, emulatedMachine=rhel6.5.0, vmType=kvm, memGuaranteedSize=1024, display=qxl, exitCode=0, pid=10456, exitReason=7, smartcardEnable=false, bootMenuEnable=false, spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,susbredir,ssmartcard, guestFQDN=, numaTune={mode=preferred}, smpCoresPerSocket=1, maxVCpus=160, exitMessage=User shut down from within the guest, clientIp=, devices=[Ljava.lang.Object;@7f860606, vmName=ffffuuu, fileTransferEnable=true, cpuType=Westmere}], log id: 2472108a >2014-07-24 09:43:52,503 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-50) [11d4a56d] Importing VM ffffuuu as external-ffffuuu, as it is running on the on Host, but does not exist in the engine. >2014-07-24 09:43:52,892 INFO [org.ovirt.engine.core.bll.AddVmFromScratchCommand] (DefaultQuartzScheduler_Worker-50) [7a179a08] Lock Acquired to object EngineLock [exclusiveLocks= key: external-ffffuuu value: VM_NAME >, sharedLocks= ] >2014-07-24 09:43:52,972 INFO [org.ovirt.engine.core.bll.AddVmFromScratchCommand] (DefaultQuartzScheduler_Worker-50) [7a179a08] Running command: AddVmFromScratchCommand internal: true. Entities affected : ID: 00000001-0001-0001-0001-0000000002c2 Type: VdsGroups >2014-07-24 09:43:53,219 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-50) [7a179a08] Correlation ID: 7a179a08, Call Stack: null, Custom Event ID: -1, Message: VM external-ffffuuu was created by Non interactive user. >2014-07-24 09:43:53,221 INFO [org.ovirt.engine.core.bll.AddVmFromScratchCommand] (DefaultQuartzScheduler_Worker-50) [7a179a08] Lock freed to object EngineLock [exclusiveLocks= key: external-ffffuuu value: VM_NAME >, sharedLocks= ] >2014-07-24 09:43:53,247 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-50) [43ad7acf] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS >2014-07-24 09:43:53,254 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-50) [17b882b7] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS >2014-07-24 09:43:53,257 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-50) [17b882b7] Host 17f7fe56-5f32-40ef-b098-611b84484e0b : rack is already in NonOperational status for reason RNG_SOURCES_INCOMPATIBLE_WITH_CLUSTER. SetNonOperationalVds command is skipped. >2014-07-24 10:03:28,532 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-40) Attempting to update VMs/Templates Ovf. >2014-07-24 10:03:28,644 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-4) The user admin from authz extension internal got updated since last interval >2014-07-24 10:27:46,095 INFO [org.ovirt.engine.core.bll.UpdateVdsGroupCommand] (ajp--127.0.0.1-8702-7) [2be7a4c0] Running command: UpdateVdsGroupCommand internal: false. Entities affected : ID: 00000001-0001-0001-0001-0000000002c2 Type: VdsGroups >2014-07-24 10:27:46,109 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-7) [2be7a4c0] Correlation ID: 2be7a4c0, Call Stack: null, Custom Event ID: -1, Message: Host cluster Default was updated by admin >2014-07-24 10:27:51,309 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (ajp--127.0.0.1-8702-1) [3659b7c9] Lock Acquired to object EngineLock [exclusiveLocks= key: 17f7fe56-5f32-40ef-b098-611b84484e0b value: VDS >, sharedLocks= ] >2014-07-24 10:27:51,346 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (org.ovirt.thread.pool-8-thread-21) [3659b7c9] Running command: ActivateVdsCommand internal: false. Entities affected : ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS >2014-07-24 10:27:51,348 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (org.ovirt.thread.pool-8-thread-21) [3659b7c9] Before acquiring lock in order to prevent monitoring for host rack from data-center Default >2014-07-24 10:27:51,349 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (org.ovirt.thread.pool-8-thread-21) [3659b7c9] Lock acquired, from now a monitoring of host will be skipped for host rack from data-center Default >2014-07-24 10:27:51,359 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-8-thread-21) [3659b7c9] START, SetVdsStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 5bfe183e >2014-07-24 10:27:51,370 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-8-thread-21) [3659b7c9] FINISH, SetVdsStatusVDSCommand, log id: 5bfe183e >2014-07-24 10:27:51,431 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (org.ovirt.thread.pool-8-thread-21) [3659b7c9] Activate finished. Lock released. Monitoring can run now for host rack from data-center Default >2014-07-24 10:27:51,444 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-21) [3659b7c9] Correlation ID: 3659b7c9, Job ID: 5d9d183f-1589-40e0-9849-9989f271eb87, Call Stack: null, Custom Event ID: -1, Message: Host rack was activated by admin. >2014-07-24 10:27:51,450 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (org.ovirt.thread.pool-8-thread-21) [3659b7c9] Lock freed to object EngineLock [exclusiveLocks= key: 17f7fe56-5f32-40ef-b098-611b84484e0b value: VDS >, sharedLocks= ] >2014-07-24 10:27:53,006 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-77) START, GetHardwareInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]), log id: 6e79d3fb >2014-07-24 10:27:53,062 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-77) FINISH, GetHardwareInfoVDSCommand, log id: 6e79d3fb >2014-07-24 10:27:53,108 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-77) [72557c4] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS >2014-07-24 10:27:53,195 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (DefaultQuartzScheduler_Worker-77) [72557c4] START, DestroyVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vmId=a2660663-ef34-41ee-a580-0fc425183c36, force=false, secondsToWait=0, gracefully=false, reason=), log id: 22cc1601 >2014-07-24 10:27:53,252 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (DefaultQuartzScheduler_Worker-77) [72557c4] FINISH, DestroyVDSCommand, log id: 22cc1601 >2014-07-24 10:27:53,287 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-77) [6c3c1f2] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-07-24 10:27:53,403 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-77) [6c3c1f2] START, SetMOMPolicyParametersVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 7f3e13e1 >2014-07-24 10:27:53,481 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-77) [6c3c1f2] FINISH, SetMOMPolicyParametersVDSCommand, log id: 7f3e13e1 >2014-07-24 10:27:53,495 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-77) [6c3c1f2] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host rack was set to Up. >2014-07-24 10:27:53,501 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-77) [6c3c1f2] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host rack. >2014-07-24 10:27:53,544 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-77) [4ab804d6] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS >2014-07-24 11:03:28,535 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-61) Attempting to update VMs/Templates Ovf. >2014-07-24 11:03:28,654 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-64) The user admin from authz extension internal got updated since last interval >2014-07-24 12:03:28,538 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-24) Attempting to update VMs/Templates Ovf. >2014-07-24 12:03:28,662 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-84) The user admin from authz extension internal got updated since last interval >2014-07-24 13:03:28,542 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-81) Attempting to update VMs/Templates Ovf. >2014-07-24 13:03:28,669 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-58) The user admin from authz extension internal got updated since last interval >2014-07-24 14:03:28,545 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-7) Attempting to update VMs/Templates Ovf. >2014-07-24 14:03:28,675 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-4) The user admin from authz extension internal got updated since last interval >2014-07-24 15:03:28,548 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-30) Attempting to update VMs/Templates Ovf. >2014-07-24 15:03:28,681 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-36) The user admin from authz extension internal got updated since last interval >2014-07-24 15:53:59,509 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp--127.0.0.1-8702-3) Running command: LoginUserCommand internal: false. >2014-07-24 15:53:59,515 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-3) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-07-24 15:53:59,530 INFO [org.ovirt.engine.core.bll.LogoutBySessionCommand] (ajp--127.0.0.1-8702-3) [19f1d247] Running command: LogoutBySessionCommand internal: false. >2014-07-24 15:53:59,534 INFO [org.ovirt.engine.core.bll.LogoutUserCommand] (ajp--127.0.0.1-8702-3) [43730762] Running command: LogoutUserCommand internal: false. >2014-07-24 15:53:59,542 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-3) [43730762] Correlation ID: 43730762, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-07-24 16:03:28,551 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-40) Attempting to update VMs/Templates Ovf. >2014-07-24 16:03:28,690 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-25) The user admin from authz extension internal got updated since last interval >2014-07-24 17:03:28,554 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-19) Attempting to update VMs/Templates Ovf. >2014-07-24 17:03:28,696 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-36) The user admin from authz extension internal got updated since last interval >2014-07-24 18:03:28,557 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-87) Attempting to update VMs/Templates Ovf. >2014-07-24 18:03:28,704 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-12) The user admin from authz extension internal got updated since last interval >2014-07-24 19:03:28,560 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-61) Attempting to update VMs/Templates Ovf. >2014-07-24 19:03:28,717 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-78) The user admin from authz extension internal got updated since last interval >2014-07-24 20:03:28,562 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-27) Attempting to update VMs/Templates Ovf. >2014-07-24 20:03:28,724 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-69) The user admin from authz extension internal got updated since last interval >2014-07-24 21:03:28,565 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-64) Attempting to update VMs/Templates Ovf. >2014-07-24 21:03:28,731 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-68) The user admin from authz extension internal got updated since last interval >2014-07-24 22:03:28,568 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-22) Attempting to update VMs/Templates Ovf. >2014-07-24 22:03:28,738 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-69) The user admin from authz extension internal got updated since last interval >2014-07-24 23:03:28,570 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-71) Attempting to update VMs/Templates Ovf. >2014-07-24 23:03:28,744 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-68) The user admin from authz extension internal got updated since last interval >2014-07-25 00:03:28,572 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-17) Attempting to update VMs/Templates Ovf. >2014-07-25 00:03:28,752 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-72) The user admin from authz extension internal got updated since last interval >2014-07-25 01:03:28,578 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-6) Attempting to update VMs/Templates Ovf. >2014-07-25 01:03:28,759 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-26) The user admin from authz extension internal got updated since last interval >2014-07-25 02:03:28,581 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-41) Attempting to update VMs/Templates Ovf. >2014-07-25 02:03:28,764 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-13) The user admin from authz extension internal got updated since last interval >2014-07-25 03:03:28,583 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-9) Attempting to update VMs/Templates Ovf. >2014-07-25 03:03:28,770 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-42) The user admin from authz extension internal got updated since last interval >2014-07-25 03:35:35,001 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-74) Start deleteAgedOutAuditLogs >2014-07-25 03:35:35,001 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-72) Start deleteAgedOutCommandEntities >2014-07-25 03:35:35,006 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-74) Finished deleteAgedOutAuditLogs >2014-07-25 03:35:35,008 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-72) Finished deleteAgedOutCommandEntities >2014-07-25 04:03:28,586 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-1) Attempting to update VMs/Templates Ovf. >2014-07-25 04:03:28,782 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-41) The user admin from authz extension internal got updated since last interval >2014-07-25 05:03:28,589 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-65) Attempting to update VMs/Templates Ovf. >2014-07-25 05:03:28,790 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-18) The user admin from authz extension internal got updated since last interval >2014-07-25 06:03:28,592 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-41) Attempting to update VMs/Templates Ovf. >2014-07-25 06:03:28,798 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-66) The user admin from authz extension internal got updated since last interval >2014-07-25 07:03:28,595 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-18) Attempting to update VMs/Templates Ovf. >2014-07-25 07:03:28,808 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-33) The user admin from authz extension internal got updated since last interval >2014-07-25 08:03:28,597 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-73) Attempting to update VMs/Templates Ovf. >2014-07-25 08:03:28,815 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-78) The user admin from authz extension internal got updated since last interval >2014-07-25 09:03:28,599 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-33) Attempting to update VMs/Templates Ovf. >2014-07-25 09:03:28,823 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-9) The user admin from authz extension internal got updated since last interval >2014-07-25 10:03:28,601 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-41) Attempting to update VMs/Templates Ovf. >2014-07-25 10:03:28,832 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-78) The user admin from authz extension internal got updated since last interval >2014-07-25 11:03:28,604 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-18) Attempting to update VMs/Templates Ovf. >2014-07-25 11:03:28,841 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-58) The user admin from authz extension internal got updated since last interval >2014-07-25 12:03:28,606 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-78) Attempting to update VMs/Templates Ovf. >2014-07-25 12:03:28,847 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-30) The user admin from authz extension internal got updated since last interval >2014-07-25 12:23:52,535 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Loaded file "/usr/share/ovirt-engine/services/ovirt-engine/ovirt-engine.conf". >2014-07-25 12:23:52,536 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) The file "/etc/ovirt-engine/engine.conf" doesn't exist or isn't readable. Will return an empty set of properties. >2014-07-25 12:23:52,537 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Loaded file "/etc/ovirt-engine/engine.conf.d/10-setup-database.conf". >2014-07-25 12:23:52,537 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Loaded file "/etc/ovirt-engine/engine.conf.d/10-setup-jboss.conf". >2014-07-25 12:23:52,552 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Loaded file "/etc/ovirt-engine/engine.conf.d/10-setup-pki.conf". >2014-07-25 12:23:52,553 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Loaded file "/etc/ovirt-engine/engine.conf.d/10-setup-protocols.conf". >2014-07-25 12:23:52,553 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_AJP_ENABLED" is "true". >2014-07-25 12:23:52,553 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_AJP_PORT" is "8702". >2014-07-25 12:23:52,554 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_APPS" is "engine.ear restapi.war legacy_restapi.war". >2014-07-25 12:23:52,554 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_CACHE" is "/var/cache/ovirt-engine". >2014-07-25 12:23:52,555 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_DB_CHECK_INTERVAL" is "1000". >2014-07-25 12:23:52,555 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_DB_CONNECTION_TIMEOUT" is "300000". >2014-07-25 12:23:52,556 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_DB_DATABASE" is "engine". >2014-07-25 12:23:52,556 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_DB_DRIVER" is "org.postgresql.Driver". >2014-07-25 12:23:52,563 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_DB_HOST" is "localhost". >2014-07-25 12:23:52,563 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_DB_MAX_CONNECTIONS" is "100". >2014-07-25 12:23:52,563 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_DB_MIN_CONNECTIONS" is "1". >2014-07-25 12:23:52,564 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_DB_PASSWORD" is "***". >2014-07-25 12:23:52,564 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_DB_PORT" is "5432". >2014-07-25 12:23:52,565 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_DB_SECURED" is "False". >2014-07-25 12:23:52,565 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_DB_SECURED_VALIDATION" is "False". >2014-07-25 12:23:52,567 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_DB_URL" is "jdbc:postgresql://localhost:5432/engine?sslfactory=org.postgresql.ssl.NonValidatingFactory". >2014-07-25 12:23:52,567 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_DB_USER" is "engine". >2014-07-25 12:23:52,568 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_DEBUG_ADDRESS" is "". >2014-07-25 12:23:52,568 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_DOC" is "/usr/share/doc/ovirt-engine". >2014-07-25 12:23:52,570 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_ETC" is "/etc/ovirt-engine". >2014-07-25 12:23:52,571 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_EXTENSION_PATH" is "/usr/share/ovirt-engine/extensions.d:/etc/ovirt-engine/extensions.d". >2014-07-25 12:23:52,572 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_FQDN" is "pb-rh35.rhev.lab.eng.brq.redhat.com". >2014-07-25 12:23:52,572 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_GROUP" is "ovirt". >2014-07-25 12:23:52,572 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_HEAP_MAX" is "1g". >2014-07-25 12:23:52,573 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_HEAP_MIN" is "1g". >2014-07-25 12:23:52,573 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_HTTPS_ENABLED" is "false". >2014-07-25 12:23:52,574 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_HTTPS_PORT" is "None". >2014-07-25 12:23:52,580 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_HTTPS_PROTOCOLS" is "SSLv3,TLSv1,TLSv1.1,TLSv1.2". >2014-07-25 12:23:52,581 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_HTTP_ENABLED" is "false". >2014-07-25 12:23:52,581 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_HTTP_PORT" is "None". >2014-07-25 12:23:52,584 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_JAVA_MODULEPATH" is "/usr/share/ovirt-engine/modules". >2014-07-25 12:23:52,584 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_JVM_ARGS" is " -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath="/var/log/ovirt-engine/dump"". >2014-07-25 12:23:52,585 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_LOG" is "/var/log/ovirt-engine". >2014-07-25 12:23:52,585 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_LOG_TO_CONSOLE" is "false". >2014-07-25 12:23:52,586 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_MANUAL" is "/usr/share/ovirt-engine/manual". >2014-07-25 12:23:52,586 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_PERM_MAX" is "256m". >2014-07-25 12:23:52,587 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_PERM_MIN" is "256m". >2014-07-25 12:23:52,587 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_PKI" is "/etc/pki/ovirt-engine". >2014-07-25 12:23:52,587 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_PKI_CA" is "/etc/pki/ovirt-engine/ca.pem". >2014-07-25 12:23:52,588 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_PKI_ENGINE_CERT" is "/etc/pki/ovirt-engine/certs/engine.cer". >2014-07-25 12:23:52,588 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_PKI_ENGINE_STORE" is "/etc/pki/ovirt-engine/keys/engine.p12". >2014-07-25 12:23:52,588 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_PKI_ENGINE_STORE_ALIAS" is "1". >2014-07-25 12:23:52,594 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_PKI_ENGINE_STORE_PASSWORD" is "***". >2014-07-25 12:23:52,594 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_PKI_TRUST_STORE" is "/etc/pki/ovirt-engine/.truststore". >2014-07-25 12:23:52,595 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_PKI_TRUST_STORE_PASSWORD" is "***". >2014-07-25 12:23:52,595 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_PROPERTIES" is " jsse.enableSNIExtension=false "java.security.krb5.conf=/etc/ovirt-engine/krb5.conf"". >2014-07-25 12:23:52,596 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_PROXY_ENABLED" is "true". >2014-07-25 12:23:52,596 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_PROXY_HTTPS_PORT" is "443". >2014-07-25 12:23:52,597 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_PROXY_HTTP_PORT" is "80". >2014-07-25 12:23:52,597 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_REPORTS_UI" is "/var/lib/ovirt-engine/reports.xml". >2014-07-25 12:23:52,598 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_STOP_INTERVAL" is "1". >2014-07-25 12:23:52,598 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_STOP_TIME" is "10". >2014-07-25 12:23:52,599 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_TMP" is "/var/tmp/ovirt-engine". >2014-07-25 12:23:52,599 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_UP_MARK" is "/var/lib/ovirt-engine/engine.up". >2014-07-25 12:23:52,603 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_URI" is "/ovirt-engine". >2014-07-25 12:23:52,604 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_USER" is "ovirt". >2014-07-25 12:23:52,604 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_USR" is "/usr/share/ovirt-engine". >2014-07-25 12:23:52,604 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_VAR" is "/var/lib/ovirt-engine". >2014-07-25 12:23:52,605 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_VERBOSE_GC" is "false". >2014-07-25 12:23:52,608 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "JBOSS_HOME" is "/usr/share/ovirt-engine-jboss-as". >2014-07-25 12:23:52,609 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "JBOSS_RUNTIME" is "/var/lib/ovirt-engine/jboss_runtime". >2014-07-25 12:23:52,612 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "SENSITIVE_KEYS" is ",ENGINE_DB_PASSWORD,ENGINE_PKI_TRUST_STORE_PASSWORD,ENGINE_PKI_ENGINE_STORE_PASSWORD". >2014-07-25 12:23:53,246 INFO [org.ovirt.engine.core.bll.Backend] (MSC service thread 1-2) Start initializing Backend >2014-07-25 12:23:53,744 INFO [org.ovirt.engine.core.utils.osinfo.OsInfoPreferencesLoader] (MSC service thread 1-2) Loaded file /etc/ovirt-engine/osinfo.conf.d/00-defaults.properties >2014-07-25 12:23:53,842 INFO [org.ovirt.engine.core.bll.Backend] (MSC service thread 1-2) Running ovirt-engine 3.5.0-0.0.master.20140722232058.git8e1babc.el6 >2014-07-25 12:23:53,843 INFO [org.ovirt.engine.core.bll.CpuFlagsManagerHandler] (MSC service thread 1-2) Start initializing dictionaries >2014-07-25 12:23:53,845 INFO [org.ovirt.engine.core.bll.CpuFlagsManagerHandler] (MSC service thread 1-2) Finished initializing dictionaries >2014-07-25 12:23:53,846 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (MSC service thread 1-2) Start initializing AuditLogCleanupManager >2014-07-25 12:23:53,847 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (MSC service thread 1-2) Setting audit cleanup manager to run at: 35 35 3 * * ? >2014-07-25 12:23:53,866 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (MSC service thread 1-2) Finished initializing AuditLogCleanupManager >2014-07-25 12:23:53,867 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (MSC service thread 1-2) Start initializing CommandEntityCleanupManager >2014-07-25 12:23:53,867 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (MSC service thread 1-2) Setting command entity cleanup manager to run at: 35 35 3 * * ? >2014-07-25 12:23:53,869 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (MSC service thread 1-2) Finished initializing CommandEntityCleanupManager >2014-07-25 12:23:53,871 INFO [org.ovirt.engine.core.bll.TagsDirector] (MSC service thread 1-2) Start initializing TagsDirector >2014-07-25 12:23:53,879 INFO [org.ovirt.engine.core.bll.TagsDirector] (MSC service thread 1-2) Tag root added to tree >2014-07-25 12:23:53,887 INFO [org.ovirt.engine.core.bll.TagsDirector] (MSC service thread 1-2) Finished initializing TagsDirector >2014-07-25 12:23:53,887 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (MSC service thread 1-2) Start initializing IsoDomainListSyncronizer >2014-07-25 12:23:53,895 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (MSC service thread 1-2) Finished initializing IsoDomainListSyncronizer >2014-07-25 12:23:53,980 INFO [org.ovirt.engine.core.bll.Backend] (MSC service thread 1-2) Completed initializing handlers >2014-07-25 12:23:54,005 INFO [org.ovirt.engine.core.utils.ErrorTranslatorImpl] (MSC service thread 1-2) Start initializing ErrorTranslatorImpl >2014-07-25 12:23:54,017 WARN [org.ovirt.engine.core.utils.ErrorTranslatorImpl] (MSC service thread 1-2) Code MAC_ADDRESS_IS_IN_USE appears more than once in string table. >2014-07-25 12:23:54,019 INFO [org.ovirt.engine.core.utils.ErrorTranslatorImpl] (MSC service thread 1-2) Finished initializing ErrorTranslatorImpl >2014-07-25 12:23:54,019 INFO [org.ovirt.engine.core.utils.ErrorTranslatorImpl] (MSC service thread 1-2) Start initializing ErrorTranslatorImpl >2014-07-25 12:23:54,020 INFO [org.ovirt.engine.core.utils.ErrorTranslatorImpl] (MSC service thread 1-2) Finished initializing ErrorTranslatorImpl >2014-07-25 12:23:54,020 INFO [org.ovirt.engine.core.bll.Backend] (MSC service thread 1-2) Mark incomplete jobs as UNKNOWN >2014-07-25 12:23:54,048 INFO [org.ovirt.engine.core.bll.job.JobRepositoryCleanupManager] (MSC service thread 1-2) Start initializing JobRepositoryCleanupManager >2014-07-25 12:23:54,049 INFO [org.ovirt.engine.core.bll.job.JobRepositoryCleanupManager] (MSC service thread 1-2) Finished initializing JobRepositoryCleanupManager >2014-07-25 12:23:54,050 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (MSC service thread 1-2) Start initializing AutoRecoveryManager >2014-07-25 12:23:54,052 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (MSC service thread 1-2) Finished initializing AutoRecoveryManager >2014-07-25 12:23:54,054 INFO [org.ovirt.engine.core.dal.job.ExecutionMessageDirector] (MSC service thread 1-2) Start initializing ExecutionMessageDirector >2014-07-25 12:23:54,056 INFO [org.ovirt.engine.core.dal.job.ExecutionMessageDirector] (MSC service thread 1-2) Finished initializing ExecutionMessageDirector >2014-07-25 12:23:54,119 INFO [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (MSC service thread 1-2) Start of enabled extensions list >2014-07-25 12:23:54,119 INFO [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (MSC service thread 1-2) Instance name: 'builtin-authn-internal', Extension name: 'Internal Authn (Built-in)', Version: 'N/A', Notes: '', License: 'ASL 2.0', Home: 'http://www.ovirt.org', Author 'The oVirt Project', Build interface Version: '0', File: 'N/A', Initialized: 'true' >2014-07-25 12:23:54,120 INFO [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (MSC service thread 1-2) Instance name: 'internal', Extension name: 'Internal Authz (Built-in)', Version: 'N/A', Notes: '', License: 'ASL 2.0', Home: 'http://www.ovirt.org', Author 'The oVirt Project', Build interface Version: '0', File: 'N/A', Initialized: 'true' >2014-07-25 12:23:54,121 INFO [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (MSC service thread 1-2) End of enabled extensions list >2014-07-25 12:23:54,197 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (MSC service thread 1-3) Start initializing DbUserCacheManager >2014-07-25 12:23:54,206 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (MSC service thread 1-3) Finished initializing DbUserCacheManager >2014-07-25 12:23:54,241 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (MSC service thread 1-3) Initialization of AsyncTaskManager completed successfully. >2014-07-25 12:23:54,244 INFO [org.ovirt.engine.core.vdsbroker.ResourceManager] (MSC service thread 1-3) Start initializing ResourceManager >2014-07-25 12:23:54,261 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-1) Principal admin::internal synchronized >2014-07-25 12:23:54,335 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (MSC service thread 1-3) Entered VdsManager constructor >2014-07-25 12:23:54,392 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (MSC service thread 1-3) Initialize vdsBroker (10.34.62.208,54,321) >2014-07-25 12:23:54,551 INFO [org.ovirt.engine.core.vdsbroker.ResourceManager] (MSC service thread 1-3) VDS 17f7fe56-5f32-40ef-b098-611b84484e0b was added to the Resource Manager >2014-07-25 12:23:54,574 INFO [org.ovirt.engine.core.vdsbroker.ResourceManager] (MSC service thread 1-3) Finished initializing ResourceManager >2014-07-25 12:23:54,579 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (MSC service thread 1-3) Initialization of OvfDataUpdater completed successfully. >2014-07-25 12:23:54,580 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (MSC service thread 1-3) Start scheduling to enable vds load balancer >2014-07-25 12:23:54,580 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (MSC service thread 1-3) Finished scheduling to enable vds load balancer >2014-07-25 12:23:54,581 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (MSC service thread 1-3) Start HA Reservation check >2014-07-25 12:23:54,582 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (MSC service thread 1-3) Finished HA Reservation check >2014-07-25 12:23:54,594 INFO [org.ovirt.engine.core.bll.InitBackendServicesOnStartupBean] (MSC service thread 1-3) Init VM custom properties utilities >2014-07-25 12:23:54,594 INFO [org.ovirt.engine.core.bll.InitBackendServicesOnStartupBean] (MSC service thread 1-3) Init device custom properties utilities >2014-07-25 12:23:54,599 INFO [org.ovirt.engine.core.bll.network.MacPoolManagerRanges] (org.ovirt.thread.pool-8-thread-1) Start initializing MacPoolManagerRanges >2014-07-25 12:23:54,604 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (MSC service thread 1-3) Initializing Scheduling manager >2014-07-25 12:23:54,619 INFO [org.ovirt.engine.core.bll.network.MacPoolManagerRanges] (org.ovirt.thread.pool-8-thread-1) Finished initializing. Available MACs in pool: 255 >2014-07-25 12:23:54,632 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (MSC service thread 1-3) External scheduler disabled, discovery skipped >2014-07-25 12:23:54,632 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (MSC service thread 1-3) Initialized Scheduling manager >2014-07-25 12:23:54,633 INFO [org.ovirt.engine.core.bll.dwh.DwhHeartBeat] (MSC service thread 1-3) Initializing DWH Heart Beat >2014-07-25 12:23:54,641 INFO [org.ovirt.engine.core.bll.dwh.DwhHeartBeat] (MSC service thread 1-3) DWH Heart Beat initialized >2014-07-25 12:23:58,416 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-6) START, GetHardwareInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]), log id: 62883ae1 >2014-07-25 12:23:58,483 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-6) FINISH, GetHardwareInfoVDSCommand, log id: 62883ae1 >2014-07-25 12:23:58,765 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-6) Initializing Host: rack >2014-07-25 12:23:58,942 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-6) [4494a716] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS >2014-07-25 12:29:09,183 INFO [org.ovirt.engine.core.bll.LoginAdminUserCommand] (ajp--127.0.0.1-8702-4) Running command: LoginAdminUserCommand internal: false. >2014-07-25 12:29:09,196 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-4) Cannot convert the string Host ${VdsName} is not responding. It will stay in Connecting state for a grace period of ${Seconds} seconds and after that an attempt to fence the host will be issued. to AuditLogType, the key does not exist in the AuditLogType declared types >2014-07-25 12:29:09,206 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-4) AuditLogType: UNASSIGNED not exist in string table >2014-07-25 12:29:09,207 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-4) AuditLogType: VDS_AUTO_FENCE_STATUS not exist in string table >2014-07-25 12:29:09,208 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-4) AuditLogType: VDS_AUTO_FENCE_STATUS_FAILED not exist in string table >2014-07-25 12:29:09,209 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-4) AuditLogType: VDS_AUTO_FENCE_FAILED_CALL_FENCE_SPM not exist in string table >2014-07-25 12:29:09,210 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-4) AuditLogType: VDS_HIGH_NETWORK_USE not exist in string table >2014-07-25 12:29:09,212 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-4) AuditLogType: USER_FAILED_REMOVE_VM not exist in string table >2014-07-25 12:29:09,213 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-4) AuditLogType: USER_RUN_UNLOCK_ENTITY_SCRIPT not exist in string table >2014-07-25 12:29:09,215 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-4) AuditLogType: VDS_NETWORK_MTU_DIFFER_FROM_LOGICAL_NETWORK not exist in string table >2014-07-25 12:29:09,216 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-4) AuditLogType: VDS_HOST_IN_CONNECTING_STATE not exist in string table >2014-07-25 12:29:09,217 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-4) AuditLogType: STORAGE_ACTIVATE_ASYNC not exist in string table >2014-07-25 12:29:09,218 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-4) AuditLogType: DWH_STOPPED not exist in string table >2014-07-25 12:29:09,219 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-4) AuditLogType: DWH_STARTED not exist in string table >2014-07-25 12:29:09,220 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-4) AuditLogType: DWH_ERROR not exist in string table >2014-07-25 12:29:09,221 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-4) AuditLogType: USER_REMOVE_AUDIT_LOG not exist in string table >2014-07-25 12:29:09,222 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-4) AuditLogType: USER_REMOVE_AUDIT_LOG_FAILED not exist in string table >2014-07-25 12:29:09,223 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-4) AuditLogType: USER_CLEAR_ALL_DISMISSED_AUDIT_LOG not exist in string table >2014-07-25 12:29:09,224 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-4) AuditLogType: USER_CLEAR_ALL_DISMISSED_AUDIT_LOG_FAILED not exist in string table >2014-07-25 12:29:09,488 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-4) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-07-25 12:29:12,631 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp--127.0.0.1-8702-2) Running command: LoginUserCommand internal: false. >2014-07-25 12:30:52,060 INFO [org.ovirt.engine.core.bll.storage.AddStorageServerConnectionCommand] (ajp--127.0.0.1-8702-2) [5f519bd8] Lock Acquired to object EngineLock [exclusiveLocks= key: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35 value: STORAGE_CONNECTION >, sharedLocks= ] >2014-07-25 12:30:52,095 INFO [org.ovirt.engine.core.bll.storage.AddStorageServerConnectionCommand] (ajp--127.0.0.1-8702-2) [5f519bd8] Running command: AddStorageServerConnectionCommand internal: false. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2014-07-25 12:30:52,109 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (ajp--127.0.0.1-8702-2) [5f519bd8] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: null, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 2973de9b >2014-07-25 12:30:52,139 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (ajp--127.0.0.1-8702-2) [5f519bd8] FINISH, ConnectStorageServerVDSCommand, return: {00000000-0000-0000-0000-000000000000=0}, log id: 2973de9b >2014-07-25 12:30:52,182 INFO [org.ovirt.engine.core.bll.storage.AddStorageServerConnectionCommand] (ajp--127.0.0.1-8702-2) [5f519bd8] Lock freed to object EngineLock [exclusiveLocks= key: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35 value: STORAGE_CONNECTION >, sharedLocks= ] >2014-07-25 12:30:52,432 INFO [org.ovirt.engine.core.bll.storage.AddNFSStorageDomainCommand] (ajp--127.0.0.1-8702-4) [35788b99] Running command: AddNFSStorageDomainCommand internal: false. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2014-07-25 12:30:52,498 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (ajp--127.0.0.1-8702-4) [35788b99] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 5454a121 >2014-07-25 12:30:52,515 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (ajp--127.0.0.1-8702-4) [35788b99] FINISH, ConnectStorageServerVDSCommand, return: {5083f370-af09-457b-9b73-3440e8b65a07=0}, log id: 5454a121 >2014-07-25 12:30:52,519 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateStorageDomainVDSCommand] (ajp--127.0.0.1-8702-4) [35788b99] START, CreateStorageDomainVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storageDomain=org.ovirt.engine.core.common.businessentities.StorageDomainStatic@89784790, args=10.34.63.202:/mnt/export/nfs/lv3/pbenas/35), log id: 6fc439b3 >2014-07-25 12:31:53,632 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateStorageDomainVDSCommand] (ajp--127.0.0.1-8702-4) [35788b99] FINISH, CreateStorageDomainVDSCommand, log id: 6fc439b3 >2014-07-25 12:31:53,637 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetStorageDomainStatsVDSCommand] (ajp--127.0.0.1-8702-4) [35788b99] START, GetStorageDomainStatsVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storageDomainId=71f82fcd-4fa6-4f56-b228-f4fa4bd4f752), log id: bbd76bf >2014-07-25 12:31:53,689 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetStorageDomainStatsVDSCommand] (ajp--127.0.0.1-8702-4) [35788b99] FINISH, GetStorageDomainStatsVDSCommand, return: org.ovirt.engine.core.common.businessentities.StorageDomain@cbe9b7bb, log id: bbd76bf >2014-07-25 12:31:53,755 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-4) [35788b99] Correlation ID: 35788b99, Job ID: 9eef15b1-c4df-4808-a278-4bc7fbf0e982, Call Stack: null, Custom Event ID: -1, Message: Storage Domain 35 was added by admin >2014-07-25 12:31:53,926 INFO [org.ovirt.engine.core.bll.storage.DisconnectStorageServerConnectionCommand] (ajp--127.0.0.1-8702-1) [b7be84c] Running command: DisconnectStorageServerConnectionCommand internal: false. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2014-07-25 12:31:53,931 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (ajp--127.0.0.1-8702-1) [b7be84c] START, DisconnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 4c915b93 >2014-07-25 12:31:54,115 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (ajp--127.0.0.1-8702-1) [b7be84c] FINISH, DisconnectStorageServerVDSCommand, return: {5083f370-af09-457b-9b73-3440e8b65a07=0}, log id: 4c915b93 >2014-07-25 12:31:54,291 INFO [org.ovirt.engine.core.bll.storage.AttachStorageDomainToPoolCommand] (ajp--127.0.0.1-8702-5) [64985a20] Running command: AttachStorageDomainToPoolCommand internal: false. Entities affected : ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage >2014-07-25 12:31:54,378 INFO [org.ovirt.engine.core.bll.storage.AddStoragePoolWithStoragesCommand] (ajp--127.0.0.1-8702-5) [1d8b9dca] Lock Acquired to object EngineLock [exclusiveLocks= key: 00000002-0002-0002-0002-0000000001d9 value: POOL >, sharedLocks= ] >2014-07-25 12:31:54,392 INFO [org.ovirt.engine.core.bll.storage.AddStoragePoolWithStoragesCommand] (ajp--127.0.0.1-8702-5) [1d8b9dca] Running command: AddStoragePoolWithStoragesCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-07-25 12:31:54,524 INFO [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (ajp--127.0.0.1-8702-5) [2afe048e] Running command: ConnectStorageToVdsCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2014-07-25 12:31:54,533 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (ajp--127.0.0.1-8702-5) [2afe048e] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 7c37d8f7 >2014-07-25 12:31:54,600 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (ajp--127.0.0.1-8702-5) [2afe048e] FINISH, ConnectStorageServerVDSCommand, return: {5083f370-af09-457b-9b73-3440e8b65a07=0}, log id: 7c37d8f7 >2014-07-25 12:31:54,608 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateStoragePoolVDSCommand] (ajp--127.0.0.1-8702-5) [2afe048e] START, CreateStoragePoolVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId=00000002-0002-0002-0002-0000000001d9, storagePoolName=Default, masterDomainId=71f82fcd-4fa6-4f56-b228-f4fa4bd4f752, domainsIdList=[71f82fcd-4fa6-4f56-b228-f4fa4bd4f752], masterVersion=1), log id: 5c6286d1 >2014-07-25 12:33:18,133 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateStoragePoolVDSCommand] (ajp--127.0.0.1-8702-5) [2afe048e] FINISH, CreateStoragePoolVDSCommand, log id: 5c6286d1 >2014-07-25 12:33:18,138 INFO [org.ovirt.engine.core.bll.storage.AddStoragePoolWithStoragesCommand] (ajp--127.0.0.1-8702-5) [2afe048e] Lock freed to object EngineLock [exclusiveLocks= key: 00000002-0002-0002-0002-0000000001d9 value: POOL >, sharedLocks= ] >2014-07-25 12:33:18,144 INFO [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (ajp--127.0.0.1-8702-5) [714136e2] Lock Acquired to object EngineLock [exclusiveLocks= key: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 value: STORAGE >, sharedLocks= ] >2014-07-25 12:33:18,150 INFO [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (ajp--127.0.0.1-8702-5) [714136e2] Running command: ActivateStorageDomainCommand internal: true. Entities affected : ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage >2014-07-25 12:33:18,170 INFO [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (ajp--127.0.0.1-8702-5) [714136e2] Lock freed to object EngineLock [exclusiveLocks= key: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 value: STORAGE >, sharedLocks= ] >2014-07-25 12:33:18,171 INFO [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (ajp--127.0.0.1-8702-5) [714136e2] ActivateStorage Domain. Before Connect all hosts to pool. Time:7/25/14 12:33 PM >2014-07-25 12:33:18,180 INFO [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-8-thread-20) [2b9d7741] Running command: ConnectStorageToVdsCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2014-07-25 12:33:18,186 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-20) [2b9d7741] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 1ba54ccc >2014-07-25 12:33:18,198 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-20) [2b9d7741] FINISH, ConnectStorageServerVDSCommand, return: {5083f370-af09-457b-9b73-3440e8b65a07=0}, log id: 1ba54ccc >2014-07-25 12:33:18,203 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ActivateStorageDomainVDSCommand] (ajp--127.0.0.1-8702-5) [714136e2] START, ActivateStorageDomainVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, storageDomainId = 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752), log id: 27f06ddd >2014-07-25 12:33:18,242 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (ajp--127.0.0.1-8702-5) [2169d51c] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-07-25 12:33:18,267 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-5) [2169d51c] Correlation ID: 2169d51c, Call Stack: null, Custom Event ID: -1, Message: Data Center is being initialized, please wait for initialization to complete. >2014-07-25 12:33:18,278 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (ajp--127.0.0.1-8702-5) [2169d51c] hostFromVds::selectedVds - rack, spmStatus Unknown_Pool, storage pool Default >2014-07-25 12:33:18,290 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (ajp--127.0.0.1-8702-5) [2169d51c] START, ConnectStoragePoolVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, masterVersion = 1), log id: 79b787a3 >2014-07-25 12:33:26,468 INFO [org.ovirt.engine.core.bll.LoginAdminUserCommand] (ajp--127.0.0.1-8702-6) Running command: LoginAdminUserCommand internal: false. >2014-07-25 12:33:26,473 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-07-25 12:33:26,542 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp--127.0.0.1-8702-7) Running command: LoginUserCommand internal: false. >2014-07-25 12:34:18,713 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (ajp--127.0.0.1-8702-5) [2169d51c] FINISH, ConnectStoragePoolVDSCommand, log id: 79b787a3 >2014-07-25 12:34:18,774 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (ajp--127.0.0.1-8702-5) [2169d51c] hostFromVds::Connected host to pool - selectedVds - rack, spmStatus Free, storage pool Default >2014-07-25 12:34:18,789 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (ajp--127.0.0.1-8702-5) [2169d51c] starting spm on vds rack, storage pool Default, prevId -1, LVER -1 >2014-07-25 12:34:18,794 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (ajp--127.0.0.1-8702-5) [2169d51c] START, SpmStartVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, prevId=-1, prevLVER=-1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 1b28ef2d >2014-07-25 12:34:18,848 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (ajp--127.0.0.1-8702-5) [2169d51c] spmStart polling started: taskId = 7ac72fd0-25e2-484b-8110-f38874cec8db >2014-07-25 12:34:19,869 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (ajp--127.0.0.1-8702-5) [2169d51c] spmStart polling ended: taskId = 7ac72fd0-25e2-484b-8110-f38874cec8db task status = finished >2014-07-25 12:34:19,926 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (ajp--127.0.0.1-8702-5) [2169d51c] spmStart polling ended, spm status: SPM >2014-07-25 12:34:19,929 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (ajp--127.0.0.1-8702-5) [2169d51c] START, HSMClearTaskVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, taskId=7ac72fd0-25e2-484b-8110-f38874cec8db), log id: 766a4f98 >2014-07-25 12:34:19,976 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (ajp--127.0.0.1-8702-5) [2169d51c] FINISH, HSMClearTaskVDSCommand, log id: 766a4f98 >2014-07-25 12:34:19,977 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (ajp--127.0.0.1-8702-5) [2169d51c] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@23d03910, log id: 1b28ef2d >2014-07-25 12:34:19,981 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (ajp--127.0.0.1-8702-5) [2169d51c] Initialize Irs proxy from vds: 10.34.62.208 >2014-07-25 12:34:19,988 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-5) [2169d51c] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208). >2014-07-25 12:34:20,085 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ActivateStorageDomainVDSCommand] (ajp--127.0.0.1-8702-5) [2169d51c] FINISH, ActivateStorageDomainVDSCommand, log id: 27f06ddd >2014-07-25 12:34:20,086 INFO [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (ajp--127.0.0.1-8702-5) [2169d51c] ActivateStorage Domain. After Connect all hosts to pool. Time:7/25/14 12:34 PM >2014-07-25 12:34:20,110 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-40) START, ConnectStoragePoolVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, masterVersion = 1), log id: 3dc9c0e6 >2014-07-25 12:34:51,876 INFO [org.ovirt.engine.core.bll.AddVmFromScratchCommand] (ajp--127.0.0.1-8702-3) [4ad76509] Lock Acquired to object EngineLock [exclusiveLocks= key: blank-disk value: VM_NAME >, sharedLocks= ] >2014-07-25 12:34:51,932 INFO [org.ovirt.engine.core.bll.AddVmFromScratchCommand] (ajp--127.0.0.1-8702-3) [4ad76509] Running command: AddVmFromScratchCommand internal: false. Entities affected : ID: 00000001-0001-0001-0001-0000000002c2 Type: VdsGroups >2014-07-25 12:34:52,150 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-3) [4ad76509] Correlation ID: 4ad76509, Job ID: fbbbb9f6-5de6-4768-81a6-e8788a231053, Call Stack: null, Custom Event ID: -1, Message: VM blank-disk was created by admin. >2014-07-25 12:34:52,162 INFO [org.ovirt.engine.core.bll.AddVmFromScratchCommand] (ajp--127.0.0.1-8702-3) [4ad76509] Lock freed to object EngineLock [exclusiveLocks= key: blank-disk value: VM_NAME >, sharedLocks= ] >2014-07-25 12:34:52,809 INFO [org.ovirt.engine.core.bll.network.vm.AddVmInterfaceCommand] (ajp--127.0.0.1-8702-6) [27b402b3] Running command: AddVmInterfaceCommand internal: false. Entities affected : ID: a69ec17d-4439-4965-84d0-5a99ccfb596b Type: VM, ID: 0000000e-000e-000e-000e-0000000003f5 Type: VnicProfile >2014-07-25 12:34:52,997 INFO [org.ovirt.engine.core.bll.network.vm.ActivateDeactivateVmNicCommand] (ajp--127.0.0.1-8702-6) [3fedec69] Running command: ActivateDeactivateVmNicCommand internal: true. Entities affected : ID: a69ec17d-4439-4965-84d0-5a99ccfb596b Type: VM >2014-07-25 12:34:53,036 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) [3fedec69] Correlation ID: 3fedec69, Call Stack: null, Custom Event ID: -1, Message: Network Interface nic1 (VirtIO) was plugged to VM blank-disk. (User: admin) >2014-07-25 12:34:53,056 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) [3fedec69] Correlation ID: 27b402b3, Call Stack: null, Custom Event ID: -1, Message: Interface nic1 (VirtIO) was added to VM blank-disk. (User: admin) >2014-07-25 12:34:53,289 INFO [org.ovirt.engine.core.bll.network.vm.ReorderVmNicsCommand] (ajp--127.0.0.1-8702-1) [330fdfc7] Running command: ReorderVmNicsCommand internal: false. Entities affected : ID: a69ec17d-4439-4965-84d0-5a99ccfb596b Type: VM >2014-07-25 12:34:55,926 INFO [org.ovirt.engine.core.bll.AddDiskCommand] (ajp--127.0.0.1-8702-7) [1c32d79d] Lock Acquired to object EngineLock [exclusiveLocks= key: a69ec17d-4439-4965-84d0-5a99ccfb596b value: VM_DISK_BOOT >, sharedLocks= key: a69ec17d-4439-4965-84d0-5a99ccfb596b value: VM >] >2014-07-25 12:34:55,961 INFO [org.ovirt.engine.core.bll.AddDiskCommand] (ajp--127.0.0.1-8702-7) [1c32d79d] Running command: AddDiskCommand internal: false. Entities affected : ID: a69ec17d-4439-4965-84d0-5a99ccfb596b Type: VM, ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage >2014-07-25 12:34:56,138 INFO [org.ovirt.engine.core.bll.AddImageFromScratchCommand] (ajp--127.0.0.1-8702-7) [1637ac6c] Running command: AddImageFromScratchCommand internal: true. Entities affected : ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage >2014-07-25 12:34:56,178 INFO [org.ovirt.engine.core.bll.AddImageFromScratchCommand] (ajp--127.0.0.1-8702-7) [1637ac6c] Lock freed to object EngineLock [exclusiveLocks= key: a69ec17d-4439-4965-84d0-5a99ccfb596b value: VM_DISK_BOOT >, sharedLocks= key: a69ec17d-4439-4965-84d0-5a99ccfb596b value: VM >] >2014-07-25 12:34:56,182 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.CreateImageVDSCommand] (ajp--127.0.0.1-8702-7) [1637ac6c] START, CreateImageVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, storageDomainId = 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752, imageGroupId = ca914369-e619-456d-8e1e-0459ff4a54cd, imageSizeInBytes = 1073741824, volumeFormat = RAW, newImageId = 1530156b-4bdd-4521-8646-af3e136ea23d, newImageDescription = ), log id: 6fdb0682 >2014-07-25 12:34:56,192 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.CreateImageVDSCommand] (ajp--127.0.0.1-8702-7) [1637ac6c] -- executeIrsBrokerCommand: calling 'createVolume' with two new parameters: description and UUID >2014-07-25 12:34:56,616 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.CreateImageVDSCommand] (ajp--127.0.0.1-8702-7) [1637ac6c] FINISH, CreateImageVDSCommand, return: 1530156b-4bdd-4521-8646-af3e136ea23d, log id: 6fdb0682 >2014-07-25 12:34:56,890 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (ajp--127.0.0.1-8702-7) [1637ac6c] CommandAsyncTask::Adding CommandMultiAsyncTasks object for command 850cd3e1-f5e5-4ce5-9516-7f03572f46a4 >2014-07-25 12:34:56,891 INFO [org.ovirt.engine.core.bll.CommandMultiAsyncTasks] (ajp--127.0.0.1-8702-7) [1637ac6c] CommandMultiAsyncTasks::AttachTask: Attaching task 030be765-71d2-41c6-ae9d-1c1d80564bc8 to command 850cd3e1-f5e5-4ce5-9516-7f03572f46a4. >2014-07-25 12:34:56,919 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (ajp--127.0.0.1-8702-7) [1637ac6c] Adding task 030be765-71d2-41c6-ae9d-1c1d80564bc8 (Parent Command AddDisk, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters), polling hasn't started yet.. >2014-07-25 12:34:56,977 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-7) [1637ac6c] Correlation ID: 1c32d79d, Job ID: 6627e838-dd49-43d2-b7cd-bb7961827f15, Call Stack: null, Custom Event ID: -1, Message: Add-Disk operation of blank-disk_Disk1 was initiated on VM blank-disk by admin. >2014-07-25 12:34:56,978 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (ajp--127.0.0.1-8702-7) [1637ac6c] BaseAsyncTask::startPollingTask: Starting to poll task 030be765-71d2-41c6-ae9d-1c1d80564bc8. >2014-07-25 12:35:04,231 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (DefaultQuartzScheduler_Worker-93) Polling and updating Async Tasks: 1 tasks, 1 tasks to poll now >2014-07-25 12:35:04,241 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (DefaultQuartzScheduler_Worker-93) SPMAsyncTask::PollTask: Polling task 030be765-71d2-41c6-ae9d-1c1d80564bc8 (Parent Command AddDisk, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) returned status finished, result 'success'. >2014-07-25 12:35:04,259 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (DefaultQuartzScheduler_Worker-93) BaseAsyncTask::OnTaskEndSuccess: Task 030be765-71d2-41c6-ae9d-1c1d80564bc8 (Parent Command AddDisk, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) ended successfully. >2014-07-25 12:35:04,263 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (DefaultQuartzScheduler_Worker-93) CommandAsyncTask::EndActionIfNecessary: All tasks of command 850cd3e1-f5e5-4ce5-9516-7f03572f46a4 has ended -> executing endAction >2014-07-25 12:35:04,265 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (DefaultQuartzScheduler_Worker-93) CommandAsyncTask::endAction: Ending action for 1 tasks (command ID: 850cd3e1-f5e5-4ce5-9516-7f03572f46a4): calling endAction . >2014-07-25 12:35:04,268 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-8-thread-5) CommandAsyncTask::EndCommandAction [within thread] context: Attempting to endAction AddDisk, executionIndex: 0 >2014-07-25 12:35:04,307 INFO [org.ovirt.engine.core.bll.AddDiskCommand] (org.ovirt.thread.pool-8-thread-5) [1c32d79d] Ending command successfully: org.ovirt.engine.core.bll.AddDiskCommand >2014-07-25 12:35:04,329 INFO [org.ovirt.engine.core.bll.AddImageFromScratchCommand] (org.ovirt.thread.pool-8-thread-5) [1637ac6c] Ending command successfully: org.ovirt.engine.core.bll.AddImageFromScratchCommand >2014-07-25 12:35:04,366 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetImageInfoVDSCommand] (org.ovirt.thread.pool-8-thread-5) [1637ac6c] START, GetImageInfoVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, storageDomainId = 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752, imageGroupId = ca914369-e619-456d-8e1e-0459ff4a54cd, imageId = 1530156b-4bdd-4521-8646-af3e136ea23d), log id: 1cb6546d >2014-07-25 12:35:04,405 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetImageInfoVDSCommand] (org.ovirt.thread.pool-8-thread-5) [1637ac6c] FINISH, GetImageInfoVDSCommand, return: org.ovirt.engine.core.common.businessentities.DiskImage@5347c20d, log id: 1cb6546d >2014-07-25 12:35:04,440 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-5) Correlation ID: 1c32d79d, Call Stack: null, Custom Event ID: -1, Message: The disk blank-disk_Disk1 was successfully added to VM blank-disk. >2014-07-25 12:35:04,442 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-8-thread-5) CommandAsyncTask::HandleEndActionResult [within thread]: endAction for action type AddDisk completed, handling the result. >2014-07-25 12:35:04,443 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-8-thread-5) CommandAsyncTask::HandleEndActionResult [within thread]: endAction for action type AddDisk succeeded, clearing tasks. >2014-07-25 12:35:04,447 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (org.ovirt.thread.pool-8-thread-5) SPMAsyncTask::ClearAsyncTask: Attempting to clear task 030be765-71d2-41c6-ae9d-1c1d80564bc8 >2014-07-25 12:35:04,450 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-5) START, SPMClearTaskVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, taskId = 030be765-71d2-41c6-ae9d-1c1d80564bc8), log id: 7c2a50d4 >2014-07-25 12:35:04,458 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-5) START, HSMClearTaskVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, taskId=030be765-71d2-41c6-ae9d-1c1d80564bc8), log id: 3e20cdfb >2014-07-25 12:35:04,527 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-5) FINISH, HSMClearTaskVDSCommand, log id: 3e20cdfb >2014-07-25 12:35:04,528 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-5) FINISH, SPMClearTaskVDSCommand, log id: 7c2a50d4 >2014-07-25 12:35:04,579 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (org.ovirt.thread.pool-8-thread-5) BaseAsyncTask::RemoveTaskFromDB: Removed task 030be765-71d2-41c6-ae9d-1c1d80564bc8 from DataBase >2014-07-25 12:35:04,580 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-8-thread-5) CommandAsyncTask::HandleEndActionResult [within thread]: Removing CommandMultiAsyncTasks object for entity 850cd3e1-f5e5-4ce5-9516-7f03572f46a4 >2014-07-25 12:35:06,987 INFO [org.ovirt.engine.core.bll.AddVmTemplateCommand] (ajp--127.0.0.1-8702-7) Running command: AddVmTemplateCommand internal: false. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-07-25 12:35:06,995 INFO [org.ovirt.engine.core.vdsbroker.SetVmStatusVDSCommand] (ajp--127.0.0.1-8702-7) START, SetVmStatusVDSCommand( vmId = a69ec17d-4439-4965-84d0-5a99ccfb596b, status = ImageLocked, exit status = Normal), log id: 4c36b646 >2014-07-25 12:35:07,009 INFO [org.ovirt.engine.core.vdsbroker.SetVmStatusVDSCommand] (ajp--127.0.0.1-8702-7) FINISH, SetVmStatusVDSCommand, log id: 4c36b646 >2014-07-25 12:35:07,042 INFO [org.ovirt.engine.core.bll.CreateImageTemplateCommand] (ajp--127.0.0.1-8702-7) [a8d66fd] Running command: CreateImageTemplateCommand internal: true. Entities affected : ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage >2014-07-25 12:35:07,061 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.CopyImageVDSCommand] (ajp--127.0.0.1-8702-7) [a8d66fd] START, CopyImageVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, storageDomainId = 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752, imageGroupId = ca914369-e619-456d-8e1e-0459ff4a54cd, imageId = 1530156b-4bdd-4521-8646-af3e136ea23d, dstImageGroupId = 4ace1a06-1fa4-4d9f-8442-fc3e2a3def19, vmId = a69ec17d-4439-4965-84d0-5a99ccfb596b, dstImageId = ba7ec5b2-dcb6-41b1-ad4c-27424db2ea00, imageDescription = Active VM, dstStorageDomainId = 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752, copyVolumeType = SharedVol, volumeFormat = RAW, preallocate = Sparse, postZero = false, force = false), log id: 69d3dbe7 >2014-07-25 12:35:07,065 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.CopyImageVDSCommand] (ajp--127.0.0.1-8702-7) [a8d66fd] -- executeIrsBrokerCommand: calling 'copyImage' with two new parameters: description and UUID >2014-07-25 12:35:07,066 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.CopyImageVDSCommand] (ajp--127.0.0.1-8702-7) [a8d66fd] -- copyImage parameters: > sdUUID=71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 > spUUID=00000002-0002-0002-0002-0000000001d9 > vmGUID=a69ec17d-4439-4965-84d0-5a99ccfb596b > srcImageGUID=ca914369-e619-456d-8e1e-0459ff4a54cd > srcVolUUID=1530156b-4bdd-4521-8646-af3e136ea23d > dstImageGUID=4ace1a06-1fa4-4d9f-8442-fc3e2a3def19 > dstVolUUID=ba7ec5b2-dcb6-41b1-ad4c-27424db2ea00 > descr=Active VM > dstSdUUID=71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 >2014-07-25 12:35:07,430 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.CopyImageVDSCommand] (ajp--127.0.0.1-8702-7) [a8d66fd] FINISH, CopyImageVDSCommand, return: 00000000-0000-0000-0000-000000000000, log id: 69d3dbe7 >2014-07-25 12:35:07,455 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (ajp--127.0.0.1-8702-7) [a8d66fd] CommandAsyncTask::Adding CommandMultiAsyncTasks object for command a0a42c45-c1b5-4aef-b4be-7addcb109965 >2014-07-25 12:35:07,456 INFO [org.ovirt.engine.core.bll.CommandMultiAsyncTasks] (ajp--127.0.0.1-8702-7) [a8d66fd] CommandMultiAsyncTasks::AttachTask: Attaching task 5e2729b7-f255-4396-ad79-5e7008d4fede to command a0a42c45-c1b5-4aef-b4be-7addcb109965. >2014-07-25 12:35:07,469 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (ajp--127.0.0.1-8702-7) [a8d66fd] Adding task 5e2729b7-f255-4396-ad79-5e7008d4fede (Parent Command AddVmTemplate, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters), polling hasn't started yet.. >2014-07-25 12:35:07,520 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-7) Correlation ID: 2e241ac8, Job ID: 44319d01-a69a-41ad-bbd1-092f24b63721, Call Stack: null, Custom Event ID: -1, Message: Creation of Template blank-disk from VM blank-disk was initiated by admin. >2014-07-25 12:35:07,522 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (ajp--127.0.0.1-8702-7) BaseAsyncTask::startPollingTask: Starting to poll task 5e2729b7-f255-4396-ad79-5e7008d4fede. >2014-07-25 12:35:14,267 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (DefaultQuartzScheduler_Worker-13) Polling and updating Async Tasks: 2 tasks, 1 tasks to poll now >2014-07-25 12:35:14,275 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (DefaultQuartzScheduler_Worker-13) SPMAsyncTask::PollTask: Polling task 5e2729b7-f255-4396-ad79-5e7008d4fede (Parent Command AddVmTemplate, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) returned status running. >2014-07-25 12:35:14,276 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (DefaultQuartzScheduler_Worker-13) Finished polling Tasks, will poll again in 10 seconds. >2014-07-25 12:35:20,359 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-40) FINISH, ConnectStoragePoolVDSCommand, log id: 3dc9c0e6 >2014-07-25 12:35:20,360 INFO [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (ajp--127.0.0.1-8702-5) [2169d51c] ActivateStorage Domain. After change storage pool status in vds. Time:7/25/14 12:35 PM >2014-07-25 12:35:20,373 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-5) [2169d51c] Correlation ID: 714136e2, Call Stack: null, Custom Event ID: -1, Message: Storage Domain 35 (Data Center Default) was activated by admin >2014-07-25 12:35:20,385 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-5) [2169d51c] Correlation ID: 1d8b9dca, Job ID: c2b818e8-7fd1-490a-aa85-4bbfcfb35b60, Call Stack: null, Custom Event ID: -1, Message: Storage Domains were attached to Data Center Default by admin >2014-07-25 12:35:20,403 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-5) [2169d51c] Correlation ID: 64985a20, Job ID: c2b818e8-7fd1-490a-aa85-4bbfcfb35b60, Call Stack: null, Custom Event ID: -1, Message: Storage Domain 35 was attached to Data Center Default by admin >2014-07-25 12:35:24,285 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (DefaultQuartzScheduler_Worker-33) SPMAsyncTask::PollTask: Polling task 5e2729b7-f255-4396-ad79-5e7008d4fede (Parent Command AddVmTemplate, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) returned status finished, result 'success'. >2014-07-25 12:35:24,292 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (DefaultQuartzScheduler_Worker-33) BaseAsyncTask::OnTaskEndSuccess: Task 5e2729b7-f255-4396-ad79-5e7008d4fede (Parent Command AddVmTemplate, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) ended successfully. >2014-07-25 12:35:24,294 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (DefaultQuartzScheduler_Worker-33) CommandAsyncTask::EndActionIfNecessary: All tasks of command a0a42c45-c1b5-4aef-b4be-7addcb109965 has ended -> executing endAction >2014-07-25 12:35:24,295 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (DefaultQuartzScheduler_Worker-33) CommandAsyncTask::endAction: Ending action for 1 tasks (command ID: a0a42c45-c1b5-4aef-b4be-7addcb109965): calling endAction . >2014-07-25 12:35:24,297 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-8-thread-19) CommandAsyncTask::EndCommandAction [within thread] context: Attempting to endAction AddVmTemplate, executionIndex: 0 >2014-07-25 12:35:24,317 INFO [org.ovirt.engine.core.bll.AddVmTemplateCommand] (org.ovirt.thread.pool-8-thread-19) [2e241ac8] Ending command successfully: org.ovirt.engine.core.bll.AddVmTemplateCommand >2014-07-25 12:35:24,321 INFO [org.ovirt.engine.core.bll.CreateImageTemplateCommand] (org.ovirt.thread.pool-8-thread-19) [a8d66fd] Ending command successfully: org.ovirt.engine.core.bll.CreateImageTemplateCommand >2014-07-25 12:35:24,324 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetImageInfoVDSCommand] (org.ovirt.thread.pool-8-thread-19) [a8d66fd] START, GetImageInfoVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, storageDomainId = 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752, imageGroupId = 4ace1a06-1fa4-4d9f-8442-fc3e2a3def19, imageId = ba7ec5b2-dcb6-41b1-ad4c-27424db2ea00), log id: 5d3c0c24 >2014-07-25 12:35:24,361 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetImageInfoVDSCommand] (org.ovirt.thread.pool-8-thread-19) [a8d66fd] FINISH, GetImageInfoVDSCommand, return: org.ovirt.engine.core.common.businessentities.DiskImage@e0cc3b16, log id: 5d3c0c24 >2014-07-25 12:35:24,368 INFO [org.ovirt.engine.core.vdsbroker.SetVmStatusVDSCommand] (org.ovirt.thread.pool-8-thread-19) START, SetVmStatusVDSCommand( vmId = a69ec17d-4439-4965-84d0-5a99ccfb596b, status = Down, exit status = Normal), log id: 780f346 >2014-07-25 12:35:24,376 INFO [org.ovirt.engine.core.vdsbroker.SetVmStatusVDSCommand] (org.ovirt.thread.pool-8-thread-19) FINISH, SetVmStatusVDSCommand, log id: 780f346 >2014-07-25 12:35:24,402 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-19) Correlation ID: 2e241ac8, Call Stack: null, Custom Event ID: -1, Message: Creation of Template blank-disk from VM blank-disk has been completed. >2014-07-25 12:35:24,403 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-8-thread-19) CommandAsyncTask::HandleEndActionResult [within thread]: endAction for action type AddVmTemplate completed, handling the result. >2014-07-25 12:35:24,404 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-8-thread-19) CommandAsyncTask::HandleEndActionResult [within thread]: endAction for action type AddVmTemplate succeeded, clearing tasks. >2014-07-25 12:35:24,409 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (org.ovirt.thread.pool-8-thread-19) SPMAsyncTask::ClearAsyncTask: Attempting to clear task 5e2729b7-f255-4396-ad79-5e7008d4fede >2014-07-25 12:35:24,410 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-19) START, SPMClearTaskVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, taskId = 5e2729b7-f255-4396-ad79-5e7008d4fede), log id: 20f87b6b >2014-07-25 12:35:24,414 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-19) START, HSMClearTaskVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, taskId=5e2729b7-f255-4396-ad79-5e7008d4fede), log id: 35aaabc5 >2014-07-25 12:35:24,474 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-19) FINISH, HSMClearTaskVDSCommand, log id: 35aaabc5 >2014-07-25 12:35:24,474 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-19) FINISH, SPMClearTaskVDSCommand, log id: 20f87b6b >2014-07-25 12:35:24,484 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (org.ovirt.thread.pool-8-thread-19) BaseAsyncTask::RemoveTaskFromDB: Removed task 5e2729b7-f255-4396-ad79-5e7008d4fede from DataBase >2014-07-25 12:35:24,485 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-8-thread-19) CommandAsyncTask::HandleEndActionResult [within thread]: Removing CommandMultiAsyncTasks object for entity a0a42c45-c1b5-4aef-b4be-7addcb109965 >2014-07-25 12:35:52,521 INFO [org.ovirt.engine.core.bll.RemoveVmCommand] (ajp--127.0.0.1-8702-6) [52c9f8ef] Lock Acquired to object EngineLock [exclusiveLocks= key: a69ec17d-4439-4965-84d0-5a99ccfb596b value: VM >, sharedLocks= ] >2014-07-25 12:35:52,562 INFO [org.ovirt.engine.core.bll.RemoveVmCommand] (org.ovirt.thread.pool-8-thread-6) [52c9f8ef] Running command: RemoveVmCommand internal: false. Entities affected : ID: a69ec17d-4439-4965-84d0-5a99ccfb596b Type: VM >2014-07-25 12:35:52,564 INFO [org.ovirt.engine.core.vdsbroker.SetVmStatusVDSCommand] (org.ovirt.thread.pool-8-thread-6) [52c9f8ef] START, SetVmStatusVDSCommand( vmId = a69ec17d-4439-4965-84d0-5a99ccfb596b, status = ImageLocked, exit status = Normal), log id: 75b3a6f6 >2014-07-25 12:35:52,571 INFO [org.ovirt.engine.core.vdsbroker.SetVmStatusVDSCommand] (org.ovirt.thread.pool-8-thread-6) [52c9f8ef] FINISH, SetVmStatusVDSCommand, log id: 75b3a6f6 >2014-07-25 12:35:52,575 INFO [org.ovirt.engine.core.bll.RemoveVmCommand] (org.ovirt.thread.pool-8-thread-6) [52c9f8ef] Lock freed to object EngineLock [exclusiveLocks= key: a69ec17d-4439-4965-84d0-5a99ccfb596b value: VM >, sharedLocks= ] >2014-07-25 12:35:52,631 INFO [org.ovirt.engine.core.bll.RemoveAllVmImagesCommand] (org.ovirt.thread.pool-8-thread-6) [20bcf485] Running command: RemoveAllVmImagesCommand internal: true. Entities affected : ID: a69ec17d-4439-4965-84d0-5a99ccfb596b Type: VM >2014-07-25 12:35:52,647 INFO [org.ovirt.engine.core.bll.RemoveImageCommand] (org.ovirt.thread.pool-8-thread-6) [5bb18aa2] Running command: RemoveImageCommand internal: true. Entities affected : ID: 00000000-0000-0000-0000-000000000000 Type: Storage >2014-07-25 12:35:52,698 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.DeleteImageGroupVDSCommand] (org.ovirt.thread.pool-8-thread-6) [5bb18aa2] START, DeleteImageGroupVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, storageDomainId = 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752, imageGroupId = ca914369-e619-456d-8e1e-0459ff4a54cd, postZeros = false, forceDelete = false), log id: a158fda >2014-07-25 12:35:53,467 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.DeleteImageGroupVDSCommand] (org.ovirt.thread.pool-8-thread-6) [5bb18aa2] FINISH, DeleteImageGroupVDSCommand, log id: a158fda >2014-07-25 12:35:53,523 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-8-thread-6) [5bb18aa2] CommandAsyncTask::Adding CommandMultiAsyncTasks object for command 9efb0336-24c0-4844-895e-ef33b3f65b8c >2014-07-25 12:35:53,525 INFO [org.ovirt.engine.core.bll.CommandMultiAsyncTasks] (org.ovirt.thread.pool-8-thread-6) [5bb18aa2] CommandMultiAsyncTasks::AttachTask: Attaching task 881c70d7-3bff-4f74-9708-93c6af989306 to command 9efb0336-24c0-4844-895e-ef33b3f65b8c. >2014-07-25 12:35:53,555 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (org.ovirt.thread.pool-8-thread-6) [5bb18aa2] Adding task 881c70d7-3bff-4f74-9708-93c6af989306 (Parent Command RemoveVm, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters), polling hasn't started yet.. >2014-07-25 12:35:53,826 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-6) [5bb18aa2] Correlation ID: 52c9f8ef, Job ID: 3e0c1f99-0524-447e-ad14-18c9e753376f, Call Stack: null, Custom Event ID: -1, Message: VM blank-disk was successfully removed. >2014-07-25 12:35:53,827 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (org.ovirt.thread.pool-8-thread-6) [5bb18aa2] BaseAsyncTask::startPollingTask: Starting to poll task 881c70d7-3bff-4f74-9708-93c6af989306. >2014-07-25 12:35:54,298 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (DefaultQuartzScheduler_Worker-1) Polling and updating Async Tasks: 3 tasks, 1 tasks to poll now >2014-07-25 12:35:54,305 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (DefaultQuartzScheduler_Worker-1) SPMAsyncTask::PollTask: Polling task 881c70d7-3bff-4f74-9708-93c6af989306 (Parent Command RemoveVm, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) returned status running. >2014-07-25 12:35:54,307 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (DefaultQuartzScheduler_Worker-1) Finished polling Tasks, will poll again in 10 seconds. >2014-07-25 12:36:04,314 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (DefaultQuartzScheduler_Worker-24) SPMAsyncTask::PollTask: Polling task 881c70d7-3bff-4f74-9708-93c6af989306 (Parent Command RemoveVm, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) returned status finished, result 'success'. >2014-07-25 12:36:04,325 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (DefaultQuartzScheduler_Worker-24) BaseAsyncTask::OnTaskEndSuccess: Task 881c70d7-3bff-4f74-9708-93c6af989306 (Parent Command RemoveVm, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) ended successfully. >2014-07-25 12:36:04,327 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (DefaultQuartzScheduler_Worker-24) CommandAsyncTask::EndActionIfNecessary: All tasks of command 9efb0336-24c0-4844-895e-ef33b3f65b8c has ended -> executing endAction >2014-07-25 12:36:04,328 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (DefaultQuartzScheduler_Worker-24) CommandAsyncTask::endAction: Ending action for 1 tasks (command ID: 9efb0336-24c0-4844-895e-ef33b3f65b8c): calling endAction . >2014-07-25 12:36:04,330 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-8-thread-16) CommandAsyncTask::EndCommandAction [within thread] context: Attempting to endAction RemoveVm, executionIndex: 0 >2014-07-25 12:36:04,343 INFO [org.ovirt.engine.core.bll.RemoveVmCommand] (org.ovirt.thread.pool-8-thread-16) Ending command successfully: org.ovirt.engine.core.bll.RemoveVmCommand >2014-07-25 12:36:04,345 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-8-thread-16) CommandAsyncTask::HandleEndActionResult [within thread]: endAction for action type RemoveVm completed, handling the result. >2014-07-25 12:36:04,346 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-8-thread-16) CommandAsyncTask::HandleEndActionResult [within thread]: endAction for action type RemoveVm succeeded, clearing tasks. >2014-07-25 12:36:04,351 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (org.ovirt.thread.pool-8-thread-16) SPMAsyncTask::ClearAsyncTask: Attempting to clear task 881c70d7-3bff-4f74-9708-93c6af989306 >2014-07-25 12:36:04,353 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-16) START, SPMClearTaskVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, taskId = 881c70d7-3bff-4f74-9708-93c6af989306), log id: 3994544b >2014-07-25 12:36:04,357 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-16) START, HSMClearTaskVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, taskId=881c70d7-3bff-4f74-9708-93c6af989306), log id: 7e0141a4 >2014-07-25 12:36:07,664 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-16) FINISH, HSMClearTaskVDSCommand, log id: 7e0141a4 >2014-07-25 12:36:07,665 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-16) FINISH, SPMClearTaskVDSCommand, log id: 3994544b >2014-07-25 12:36:07,685 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (org.ovirt.thread.pool-8-thread-16) BaseAsyncTask::RemoveTaskFromDB: Removed task 881c70d7-3bff-4f74-9708-93c6af989306 from DataBase >2014-07-25 12:36:07,686 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-8-thread-16) CommandAsyncTask::HandleEndActionResult [within thread]: Removing CommandMultiAsyncTasks object for entity 9efb0336-24c0-4844-895e-ef33b3f65b8c >2014-07-25 12:36:23,644 INFO [org.ovirt.engine.core.bll.AddVmFromTemplateCommand] (ajp--127.0.0.1-8702-3) [1ab40ee9] Lock Acquired to object EngineLock [exclusiveLocks= key: runit value: VM_NAME >, sharedLocks= key: d031b0a2-efae-4300-b703-af169651be32 value: TEMPLATE >key: 4ace1a06-1fa4-4d9f-8442-fc3e2a3def19 value: DISK >] >2014-07-25 12:36:23,709 INFO [org.ovirt.engine.core.bll.AddVmFromTemplateCommand] (ajp--127.0.0.1-8702-3) [1ab40ee9] Running command: AddVmFromTemplateCommand internal: false. Entities affected : ID: 00000001-0001-0001-0001-0000000002c2 Type: VdsGroups, ID: d031b0a2-efae-4300-b703-af169651be32 Type: VmTemplate, ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage >2014-07-25 12:36:23,745 INFO [org.ovirt.engine.core.vdsbroker.SetVmStatusVDSCommand] (ajp--127.0.0.1-8702-3) [1ab40ee9] START, SetVmStatusVDSCommand( vmId = f7613d6f-8091-4d21-90e6-c13219a1b3db, status = ImageLocked, exit status = Normal), log id: 56d513e9 >2014-07-25 12:36:23,750 INFO [org.ovirt.engine.core.vdsbroker.SetVmStatusVDSCommand] (ajp--127.0.0.1-8702-3) [1ab40ee9] FINISH, SetVmStatusVDSCommand, log id: 56d513e9 >2014-07-25 12:36:23,885 INFO [org.ovirt.engine.core.bll.CreateCloneOfTemplateCommand] (ajp--127.0.0.1-8702-3) [76f353cc] Running command: CreateCloneOfTemplateCommand internal: true. Entities affected : ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage >2014-07-25 12:36:23,971 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.CopyImageVDSCommand] (ajp--127.0.0.1-8702-3) [76f353cc] START, CopyImageVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, storageDomainId = 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752, imageGroupId = 4ace1a06-1fa4-4d9f-8442-fc3e2a3def19, imageId = ba7ec5b2-dcb6-41b1-ad4c-27424db2ea00, dstImageGroupId = 1f7ff79b-b034-47ad-8864-181be4e1de5a, vmId = d031b0a2-efae-4300-b703-af169651be32, dstImageId = 827408c6-d52c-4c49-92f6-461ce222e485, imageDescription = , dstStorageDomainId = 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752, copyVolumeType = LeafVol, volumeFormat = RAW, preallocate = Sparse, postZero = false, force = false), log id: 75999916 >2014-07-25 12:36:23,975 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.CopyImageVDSCommand] (ajp--127.0.0.1-8702-3) [76f353cc] -- executeIrsBrokerCommand: calling 'copyImage' with two new parameters: description and UUID >2014-07-25 12:36:23,976 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.CopyImageVDSCommand] (ajp--127.0.0.1-8702-3) [76f353cc] -- copyImage parameters: > sdUUID=71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 > spUUID=00000002-0002-0002-0002-0000000001d9 > vmGUID=d031b0a2-efae-4300-b703-af169651be32 > srcImageGUID=4ace1a06-1fa4-4d9f-8442-fc3e2a3def19 > srcVolUUID=ba7ec5b2-dcb6-41b1-ad4c-27424db2ea00 > dstImageGUID=1f7ff79b-b034-47ad-8864-181be4e1de5a > dstVolUUID=827408c6-d52c-4c49-92f6-461ce222e485 > descr= > dstSdUUID=71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 >2014-07-25 12:36:24,227 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (DefaultQuartzScheduler_Worker-66) Setting new tasks map. The map contains now 2 tasks >2014-07-25 12:36:25,720 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.CopyImageVDSCommand] (ajp--127.0.0.1-8702-3) [76f353cc] FINISH, CopyImageVDSCommand, return: 00000000-0000-0000-0000-000000000000, log id: 75999916 >2014-07-25 12:36:25,760 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (ajp--127.0.0.1-8702-3) [76f353cc] CommandAsyncTask::Adding CommandMultiAsyncTasks object for command 37d356ed-5485-42b9-b009-43ec0801ae2b >2014-07-25 12:36:25,761 INFO [org.ovirt.engine.core.bll.CommandMultiAsyncTasks] (ajp--127.0.0.1-8702-3) [76f353cc] CommandMultiAsyncTasks::AttachTask: Attaching task c3598f13-53e7-408c-8e69-ec66e386df0f to command 37d356ed-5485-42b9-b009-43ec0801ae2b. >2014-07-25 12:36:25,802 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (ajp--127.0.0.1-8702-3) [76f353cc] Adding task c3598f13-53e7-408c-8e69-ec66e386df0f (Parent Command AddVmFromTemplate, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters), polling hasn't started yet.. >2014-07-25 12:36:25,904 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-3) [76f353cc] Correlation ID: 1ab40ee9, Job ID: 50b98694-8560-4821-b756-9c4d4e038a15, Call Stack: null, Custom Event ID: -1, Message: VM runit creation was initiated by admin. >2014-07-25 12:36:25,906 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (ajp--127.0.0.1-8702-3) [76f353cc] BaseAsyncTask::startPollingTask: Starting to poll task c3598f13-53e7-408c-8e69-ec66e386df0f. >2014-07-25 12:36:26,375 INFO [org.ovirt.engine.core.bll.network.vm.ReorderVmNicsCommand] (ajp--127.0.0.1-8702-6) [3de12490] Running command: ReorderVmNicsCommand internal: false. Entities affected : ID: f7613d6f-8091-4d21-90e6-c13219a1b3db Type: VM >2014-07-25 12:36:34,336 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (DefaultQuartzScheduler_Worker-92) Polling and updating Async Tasks: 3 tasks, 1 tasks to poll now >2014-07-25 12:36:34,344 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (DefaultQuartzScheduler_Worker-92) SPMAsyncTask::PollTask: Polling task c3598f13-53e7-408c-8e69-ec66e386df0f (Parent Command AddVmFromTemplate, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) returned status running. >2014-07-25 12:36:34,345 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (DefaultQuartzScheduler_Worker-92) Finished polling Tasks, will poll again in 10 seconds. >2014-07-25 12:36:44,353 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (DefaultQuartzScheduler_Worker-13) SPMAsyncTask::PollTask: Polling task c3598f13-53e7-408c-8e69-ec66e386df0f (Parent Command AddVmFromTemplate, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) returned status finished, result 'success'. >2014-07-25 12:36:44,362 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (DefaultQuartzScheduler_Worker-13) BaseAsyncTask::OnTaskEndSuccess: Task c3598f13-53e7-408c-8e69-ec66e386df0f (Parent Command AddVmFromTemplate, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) ended successfully. >2014-07-25 12:36:44,371 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (DefaultQuartzScheduler_Worker-13) CommandAsyncTask::EndActionIfNecessary: All tasks of command 37d356ed-5485-42b9-b009-43ec0801ae2b has ended -> executing endAction >2014-07-25 12:36:44,373 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (DefaultQuartzScheduler_Worker-13) CommandAsyncTask::endAction: Ending action for 1 tasks (command ID: 37d356ed-5485-42b9-b009-43ec0801ae2b): calling endAction . >2014-07-25 12:36:44,374 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-8-thread-14) CommandAsyncTask::EndCommandAction [within thread] context: Attempting to endAction AddVmFromTemplate, executionIndex: 0 >2014-07-25 12:36:44,420 INFO [org.ovirt.engine.core.bll.AddVmFromTemplateCommand] (org.ovirt.thread.pool-8-thread-14) [1ab40ee9] Ending command successfully: org.ovirt.engine.core.bll.AddVmFromTemplateCommand >2014-07-25 12:36:44,434 INFO [org.ovirt.engine.core.bll.CreateSnapshotFromTemplateCommand] (org.ovirt.thread.pool-8-thread-14) [76f353cc] Ending command successfully: org.ovirt.engine.core.bll.CreateSnapshotFromTemplateCommand >2014-07-25 12:36:44,442 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetImageInfoVDSCommand] (org.ovirt.thread.pool-8-thread-14) [76f353cc] START, GetImageInfoVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, storageDomainId = 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752, imageGroupId = 1f7ff79b-b034-47ad-8864-181be4e1de5a, imageId = 827408c6-d52c-4c49-92f6-461ce222e485), log id: 27b3c6c3 >2014-07-25 12:36:44,476 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetImageInfoVDSCommand] (org.ovirt.thread.pool-8-thread-14) [76f353cc] FINISH, GetImageInfoVDSCommand, return: org.ovirt.engine.core.common.businessentities.DiskImage@2a00c8d9, log id: 27b3c6c3 >2014-07-25 12:36:44,483 INFO [org.ovirt.engine.core.vdsbroker.SetVmStatusVDSCommand] (org.ovirt.thread.pool-8-thread-14) START, SetVmStatusVDSCommand( vmId = f7613d6f-8091-4d21-90e6-c13219a1b3db, status = Down, exit status = Normal), log id: 12c5b010 >2014-07-25 12:36:44,488 INFO [org.ovirt.engine.core.vdsbroker.SetVmStatusVDSCommand] (org.ovirt.thread.pool-8-thread-14) FINISH, SetVmStatusVDSCommand, log id: 12c5b010 >2014-07-25 12:36:44,491 INFO [org.ovirt.engine.core.bll.AddVmFromTemplateCommand] (org.ovirt.thread.pool-8-thread-14) Lock freed to object EngineLock [exclusiveLocks= key: runit value: VM_NAME >, sharedLocks= key: d031b0a2-efae-4300-b703-af169651be32 value: TEMPLATE >key: 4ace1a06-1fa4-4d9f-8442-fc3e2a3def19 value: DISK >] >2014-07-25 12:36:44,502 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-14) Correlation ID: 1ab40ee9, Call Stack: null, Custom Event ID: -1, Message: VM runit creation has been completed. >2014-07-25 12:36:44,503 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-8-thread-14) CommandAsyncTask::HandleEndActionResult [within thread]: endAction for action type AddVmFromTemplate completed, handling the result. >2014-07-25 12:36:44,505 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-8-thread-14) CommandAsyncTask::HandleEndActionResult [within thread]: endAction for action type AddVmFromTemplate succeeded, clearing tasks. >2014-07-25 12:36:44,510 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (org.ovirt.thread.pool-8-thread-14) SPMAsyncTask::ClearAsyncTask: Attempting to clear task c3598f13-53e7-408c-8e69-ec66e386df0f >2014-07-25 12:36:44,511 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-14) START, SPMClearTaskVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, taskId = c3598f13-53e7-408c-8e69-ec66e386df0f), log id: 7d6b229e >2014-07-25 12:36:44,514 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-14) START, HSMClearTaskVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, taskId=c3598f13-53e7-408c-8e69-ec66e386df0f), log id: 4adf366 >2014-07-25 12:36:44,571 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-14) FINISH, HSMClearTaskVDSCommand, log id: 4adf366 >2014-07-25 12:36:44,571 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-14) FINISH, SPMClearTaskVDSCommand, log id: 7d6b229e >2014-07-25 12:36:44,586 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (org.ovirt.thread.pool-8-thread-14) BaseAsyncTask::RemoveTaskFromDB: Removed task c3598f13-53e7-408c-8e69-ec66e386df0f from DataBase >2014-07-25 12:36:44,587 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-8-thread-14) CommandAsyncTask::HandleEndActionResult [within thread]: Removing CommandMultiAsyncTasks object for entity 37d356ed-5485-42b9-b009-43ec0801ae2b >2014-07-25 12:36:48,727 INFO [org.ovirt.engine.core.bll.AddVmFromScratchCommand] (ajp--127.0.0.1-8702-3) [35bad99] Lock Acquired to object EngineLock [exclusiveLocks= key: new-one value: VM_NAME >, sharedLocks= ] >2014-07-25 12:36:48,764 INFO [org.ovirt.engine.core.bll.AddVmFromScratchCommand] (ajp--127.0.0.1-8702-3) [35bad99] Running command: AddVmFromScratchCommand internal: false. Entities affected : ID: 00000001-0001-0001-0001-0000000002c2 Type: VdsGroups >2014-07-25 12:36:48,833 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-3) [35bad99] Correlation ID: 35bad99, Job ID: dc49a405-5464-4301-91b7-828c687b2d7b, Call Stack: null, Custom Event ID: -1, Message: VM new-one was created by admin. >2014-07-25 12:36:48,839 INFO [org.ovirt.engine.core.bll.AddVmFromScratchCommand] (ajp--127.0.0.1-8702-3) [35bad99] Lock freed to object EngineLock [exclusiveLocks= key: new-one value: VM_NAME >, sharedLocks= ] >2014-07-25 12:36:49,418 INFO [org.ovirt.engine.core.bll.network.vm.AddVmInterfaceCommand] (ajp--127.0.0.1-8702-4) [54bc166a] Running command: AddVmInterfaceCommand internal: false. Entities affected : ID: 6f56ce08-46a9-4bfe-b207-50fd71e003cc Type: VM, ID: 0000000e-000e-000e-000e-0000000003f5 Type: VnicProfile >2014-07-25 12:36:49,448 INFO [org.ovirt.engine.core.bll.network.vm.ActivateDeactivateVmNicCommand] (ajp--127.0.0.1-8702-4) [34d4de61] Running command: ActivateDeactivateVmNicCommand internal: true. Entities affected : ID: 6f56ce08-46a9-4bfe-b207-50fd71e003cc Type: VM >2014-07-25 12:36:49,468 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-4) [34d4de61] Correlation ID: 34d4de61, Call Stack: null, Custom Event ID: -1, Message: Network Interface nic1 (VirtIO) was plugged to VM new-one. (User: admin) >2014-07-25 12:36:49,478 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-4) [34d4de61] Correlation ID: 54bc166a, Call Stack: null, Custom Event ID: -1, Message: Interface nic1 (VirtIO) was added to VM new-one. (User: admin) >2014-07-25 12:36:49,570 INFO [org.ovirt.engine.core.bll.network.vm.ReorderVmNicsCommand] (ajp--127.0.0.1-8702-6) [5c934e7c] Running command: ReorderVmNicsCommand internal: false. Entities affected : ID: 6f56ce08-46a9-4bfe-b207-50fd71e003cc Type: VM >2014-07-25 12:36:52,489 INFO [org.ovirt.engine.core.bll.AddDiskCommand] (ajp--127.0.0.1-8702-2) [a186681] Lock Acquired to object EngineLock [exclusiveLocks= key: 6f56ce08-46a9-4bfe-b207-50fd71e003cc value: VM_DISK_BOOT >, sharedLocks= key: 6f56ce08-46a9-4bfe-b207-50fd71e003cc value: VM >] >2014-07-25 12:36:52,516 INFO [org.ovirt.engine.core.bll.AddDiskCommand] (ajp--127.0.0.1-8702-2) [a186681] Running command: AddDiskCommand internal: false. Entities affected : ID: 6f56ce08-46a9-4bfe-b207-50fd71e003cc Type: VM, ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage >2014-07-25 12:36:52,545 INFO [org.ovirt.engine.core.bll.AddImageFromScratchCommand] (ajp--127.0.0.1-8702-2) [1338c46b] Running command: AddImageFromScratchCommand internal: true. Entities affected : ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage >2014-07-25 12:36:52,570 INFO [org.ovirt.engine.core.bll.AddImageFromScratchCommand] (ajp--127.0.0.1-8702-2) [1338c46b] Lock freed to object EngineLock [exclusiveLocks= key: 6f56ce08-46a9-4bfe-b207-50fd71e003cc value: VM_DISK_BOOT >, sharedLocks= key: 6f56ce08-46a9-4bfe-b207-50fd71e003cc value: VM >] >2014-07-25 12:36:52,574 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.CreateImageVDSCommand] (ajp--127.0.0.1-8702-2) [1338c46b] START, CreateImageVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, storageDomainId = 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752, imageGroupId = 43a334b9-1e3f-472c-b32e-e1ed0c074c14, imageSizeInBytes = 1073741824, volumeFormat = RAW, newImageId = 860b11ae-d1a7-4c83-ad10-4989fcca0638, newImageDescription = ), log id: 1487e536 >2014-07-25 12:36:52,579 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.CreateImageVDSCommand] (ajp--127.0.0.1-8702-2) [1338c46b] -- executeIrsBrokerCommand: calling 'createVolume' with two new parameters: description and UUID >2014-07-25 12:36:54,228 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (DefaultQuartzScheduler_Worker-33) Setting new tasks map. The map contains now 2 tasks >2014-07-25 12:36:59,133 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.CreateImageVDSCommand] (ajp--127.0.0.1-8702-2) [1338c46b] FINISH, CreateImageVDSCommand, return: 860b11ae-d1a7-4c83-ad10-4989fcca0638, log id: 1487e536 >2014-07-25 12:36:59,142 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (ajp--127.0.0.1-8702-2) [1338c46b] CommandAsyncTask::Adding CommandMultiAsyncTasks object for command fec024fe-8cd6-4fbc-921c-2ff9dcc97dd7 >2014-07-25 12:36:59,145 INFO [org.ovirt.engine.core.bll.CommandMultiAsyncTasks] (ajp--127.0.0.1-8702-2) [1338c46b] CommandMultiAsyncTasks::AttachTask: Attaching task 7ac7f77c-031c-4313-b1ee-0acaf9345c00 to command fec024fe-8cd6-4fbc-921c-2ff9dcc97dd7. >2014-07-25 12:36:59,177 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (ajp--127.0.0.1-8702-2) [1338c46b] Adding task 7ac7f77c-031c-4313-b1ee-0acaf9345c00 (Parent Command AddDisk, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters), polling hasn't started yet.. >2014-07-25 12:36:59,208 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-2) [1338c46b] Correlation ID: a186681, Job ID: c446e85e-37e1-41fd-8915-a9c0eff06428, Call Stack: null, Custom Event ID: -1, Message: Add-Disk operation of new-one_Disk1 was initiated on VM new-one by admin. >2014-07-25 12:36:59,209 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (ajp--127.0.0.1-8702-2) [1338c46b] BaseAsyncTask::startPollingTask: Starting to poll task 7ac7f77c-031c-4313-b1ee-0acaf9345c00. >2014-07-25 12:37:04,376 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (DefaultQuartzScheduler_Worker-58) Polling and updating Async Tasks: 3 tasks, 1 tasks to poll now >2014-07-25 12:37:04,383 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (DefaultQuartzScheduler_Worker-58) SPMAsyncTask::PollTask: Polling task 7ac7f77c-031c-4313-b1ee-0acaf9345c00 (Parent Command AddDisk, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) returned status running. >2014-07-25 12:37:04,386 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (DefaultQuartzScheduler_Worker-58) Finished polling Tasks, will poll again in 10 seconds. >2014-07-25 12:37:05,333 INFO [org.ovirt.engine.core.bll.RunVmCommand] (ajp--127.0.0.1-8702-8) [2abfccd] Lock Acquired to object EngineLock [exclusiveLocks= key: 6f56ce08-46a9-4bfe-b207-50fd71e003cc value: VM >, sharedLocks= ] >2014-07-25 12:37:05,385 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (ajp--127.0.0.1-8702-8) [2abfccd] START, IsVmDuringInitiatingVDSCommand( vmId = 6f56ce08-46a9-4bfe-b207-50fd71e003cc), log id: 519a9fe8 >2014-07-25 12:37:05,386 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (ajp--127.0.0.1-8702-8) [2abfccd] FINISH, IsVmDuringInitiatingVDSCommand, return: false, log id: 519a9fe8 >2014-07-25 12:37:05,390 WARN [org.ovirt.engine.core.bll.RunVmCommand] (ajp--127.0.0.1-8702-8) [2abfccd] CanDoAction of action RunVm failed. Reasons:VAR__ACTION__RUN,VAR__TYPE__VM,ACTION_TYPE_FAILED_DISKS_LOCKED,$diskAliases new-one_Disk1 >2014-07-25 12:37:05,391 INFO [org.ovirt.engine.core.bll.RunVmCommand] (ajp--127.0.0.1-8702-8) [2abfccd] Lock freed to object EngineLock [exclusiveLocks= key: 6f56ce08-46a9-4bfe-b207-50fd71e003cc value: VM >, sharedLocks= ] >2014-07-25 12:37:13,163 INFO [org.ovirt.engine.core.bll.RunVmCommand] (ajp--127.0.0.1-8702-7) [85e122a] Lock Acquired to object EngineLock [exclusiveLocks= key: 6f56ce08-46a9-4bfe-b207-50fd71e003cc value: VM >, sharedLocks= ] >2014-07-25 12:37:13,197 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (ajp--127.0.0.1-8702-7) [85e122a] START, IsVmDuringInitiatingVDSCommand( vmId = 6f56ce08-46a9-4bfe-b207-50fd71e003cc), log id: 2f8fc05b >2014-07-25 12:37:13,199 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (ajp--127.0.0.1-8702-7) [85e122a] FINISH, IsVmDuringInitiatingVDSCommand, return: false, log id: 2f8fc05b >2014-07-25 12:37:13,202 WARN [org.ovirt.engine.core.bll.RunVmCommand] (ajp--127.0.0.1-8702-7) [85e122a] CanDoAction of action RunVm failed. Reasons:VAR__ACTION__RUN,VAR__TYPE__VM,ACTION_TYPE_FAILED_DISKS_LOCKED,$diskAliases new-one_Disk1 >2014-07-25 12:37:13,203 INFO [org.ovirt.engine.core.bll.RunVmCommand] (ajp--127.0.0.1-8702-7) [85e122a] Lock freed to object EngineLock [exclusiveLocks= key: 6f56ce08-46a9-4bfe-b207-50fd71e003cc value: VM >, sharedLocks= ] >2014-07-25 12:37:14,393 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (DefaultQuartzScheduler_Worker-78) SPMAsyncTask::PollTask: Polling task 7ac7f77c-031c-4313-b1ee-0acaf9345c00 (Parent Command AddDisk, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) returned status finished, result 'success'. >2014-07-25 12:37:14,401 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (DefaultQuartzScheduler_Worker-78) BaseAsyncTask::OnTaskEndSuccess: Task 7ac7f77c-031c-4313-b1ee-0acaf9345c00 (Parent Command AddDisk, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) ended successfully. >2014-07-25 12:37:14,403 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (DefaultQuartzScheduler_Worker-78) CommandAsyncTask::EndActionIfNecessary: All tasks of command fec024fe-8cd6-4fbc-921c-2ff9dcc97dd7 has ended -> executing endAction >2014-07-25 12:37:14,405 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (DefaultQuartzScheduler_Worker-78) CommandAsyncTask::endAction: Ending action for 1 tasks (command ID: fec024fe-8cd6-4fbc-921c-2ff9dcc97dd7): calling endAction . >2014-07-25 12:37:14,406 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-8-thread-7) CommandAsyncTask::EndCommandAction [within thread] context: Attempting to endAction AddDisk, executionIndex: 0 >2014-07-25 12:37:14,417 INFO [org.ovirt.engine.core.bll.AddDiskCommand] (org.ovirt.thread.pool-8-thread-7) [a186681] Ending command successfully: org.ovirt.engine.core.bll.AddDiskCommand >2014-07-25 12:37:14,435 INFO [org.ovirt.engine.core.bll.AddImageFromScratchCommand] (org.ovirt.thread.pool-8-thread-7) [1338c46b] Ending command successfully: org.ovirt.engine.core.bll.AddImageFromScratchCommand >2014-07-25 12:37:14,460 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetImageInfoVDSCommand] (org.ovirt.thread.pool-8-thread-7) [1338c46b] START, GetImageInfoVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, storageDomainId = 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752, imageGroupId = 43a334b9-1e3f-472c-b32e-e1ed0c074c14, imageId = 860b11ae-d1a7-4c83-ad10-4989fcca0638), log id: 21e9968d >2014-07-25 12:37:14,494 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetImageInfoVDSCommand] (org.ovirt.thread.pool-8-thread-7) [1338c46b] FINISH, GetImageInfoVDSCommand, return: org.ovirt.engine.core.common.businessentities.DiskImage@e991043e, log id: 21e9968d >2014-07-25 12:37:14,513 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-7) Correlation ID: a186681, Call Stack: null, Custom Event ID: -1, Message: The disk new-one_Disk1 was successfully added to VM new-one. >2014-07-25 12:37:14,514 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-8-thread-7) CommandAsyncTask::HandleEndActionResult [within thread]: endAction for action type AddDisk completed, handling the result. >2014-07-25 12:37:14,516 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-8-thread-7) CommandAsyncTask::HandleEndActionResult [within thread]: endAction for action type AddDisk succeeded, clearing tasks. >2014-07-25 12:37:14,520 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (org.ovirt.thread.pool-8-thread-7) SPMAsyncTask::ClearAsyncTask: Attempting to clear task 7ac7f77c-031c-4313-b1ee-0acaf9345c00 >2014-07-25 12:37:14,521 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-7) START, SPMClearTaskVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, taskId = 7ac7f77c-031c-4313-b1ee-0acaf9345c00), log id: 34d9da9f >2014-07-25 12:37:14,531 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-7) START, HSMClearTaskVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, taskId=7ac7f77c-031c-4313-b1ee-0acaf9345c00), log id: 4aa017d4 >2014-07-25 12:37:14,581 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-7) FINISH, HSMClearTaskVDSCommand, log id: 4aa017d4 >2014-07-25 12:37:14,582 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-7) FINISH, SPMClearTaskVDSCommand, log id: 34d9da9f >2014-07-25 12:37:14,594 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (org.ovirt.thread.pool-8-thread-7) BaseAsyncTask::RemoveTaskFromDB: Removed task 7ac7f77c-031c-4313-b1ee-0acaf9345c00 from DataBase >2014-07-25 12:37:14,596 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-8-thread-7) CommandAsyncTask::HandleEndActionResult [within thread]: Removing CommandMultiAsyncTasks object for entity fec024fe-8cd6-4fbc-921c-2ff9dcc97dd7 >2014-07-25 12:37:24,004 INFO [org.ovirt.engine.core.bll.RunVmCommand] (ajp--127.0.0.1-8702-8) [48ba118e] Lock Acquired to object EngineLock [exclusiveLocks= key: 6f56ce08-46a9-4bfe-b207-50fd71e003cc value: VM >, sharedLocks= ] >2014-07-25 12:37:24,107 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (ajp--127.0.0.1-8702-8) [48ba118e] START, IsVmDuringInitiatingVDSCommand( vmId = 6f56ce08-46a9-4bfe-b207-50fd71e003cc), log id: 3d6eef82 >2014-07-25 12:37:24,108 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (ajp--127.0.0.1-8702-8) [48ba118e] FINISH, IsVmDuringInitiatingVDSCommand, return: false, log id: 3d6eef82 >2014-07-25 12:37:24,232 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (DefaultQuartzScheduler_Worker-100) Setting new tasks map. The map contains now 2 tasks >2014-07-25 12:37:24,318 INFO [org.ovirt.engine.core.bll.RunVmCommand] (org.ovirt.thread.pool-8-thread-14) [48ba118e] Running command: RunVmCommand internal: false. Entities affected : ID: 6f56ce08-46a9-4bfe-b207-50fd71e003cc Type: VM >2014-07-25 12:37:24,510 INFO [org.ovirt.engine.core.vdsbroker.UpdateVmDynamicDataVDSCommand] (org.ovirt.thread.pool-8-thread-14) [48ba118e] START, UpdateVmDynamicDataVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vmDynamic=org.ovirt.engine.core.common.businessentities.VmDynamic@e46273a5), log id: 7e7740f >2014-07-25 12:37:24,516 INFO [org.ovirt.engine.core.vdsbroker.UpdateVmDynamicDataVDSCommand] (org.ovirt.thread.pool-8-thread-14) [48ba118e] FINISH, UpdateVmDynamicDataVDSCommand, log id: 7e7740f >2014-07-25 12:37:24,535 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (org.ovirt.thread.pool-8-thread-14) [48ba118e] START, CreateVmVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vmId=6f56ce08-46a9-4bfe-b207-50fd71e003cc, vm=VM [new-one]), log id: 1498ae3e >2014-07-25 12:37:24,642 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVmFromCloudInitVDSCommand] (org.ovirt.thread.pool-8-thread-14) [48ba118e] START, CreateVmFromCloudInitVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vmId=6f56ce08-46a9-4bfe-b207-50fd71e003cc, vm=VM [new-one]), log id: 598c9d44 >2014-07-25 12:37:24,756 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (org.ovirt.thread.pool-8-thread-14) [48ba118e] org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVmFromCloudInitVDSCommand kvmEnable=true,keyboardLayout=en-us,nice=0,displayNetwork=ovirtmgmt,copyPasteEnable=true,timeOffset=0,transparentHugePages=true,vmId=6f56ce08-46a9-4bfe-b207-50fd71e003cc,acpiEnable=true,custom={},spiceSslCipherSuite=DEFAULT,memSize=1024,smp=1,emulatedMachine=rhel6.5.0,vmType=kvm,memGuaranteedSize=1024,display=qxl,smartcardEnable=false,bootMenuEnable=false,spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,susbredir,ssmartcard,smpCoresPerSocket=1,numaTune={mode=preferred},maxVCpus=160,devices=[{shared=false, iface=ide, index=3, specParams={vmPayload={file={openstack/latest/meta_data.json=ewogICJsYXVuY2hfaW5kZXgiIDogIjAiLAogICJhdmFpbGFiaWxpdHlfem9uZSIgOiAibm92YSIs >CiAgIm5ldHdvcmstaW50ZXJmYWNlcyIgOiAiYXV0b1xuIiwKICAibmV0d29ya19jb25maWciIDog >ewogICAgImNvbnRlbnRfcGF0aCIgOiAiL2NvbnRlbnQvMDAwMCIsCiAgICAicGF0aCIgOiAiL2V0 >Yy9uZXR3b3JrL2ludGVyZmFjZXMiCiAgfSwKICAidXVpZCIgOiAiMTI0M2Y4ZjgtN2U3MS00NTFj >LWI1M2QtYWViZmJiOWQyOThlIiwKICAibWV0YSIgOiB7CiAgICAiZXNzZW50aWFsIiA6ICJmYWxz >ZSIsCiAgICAicm9sZSIgOiAic2VydmVyIiwKICAgICJkc21vZGUiIDogImxvY2FsIgogIH0KfQ== >, openstack/latest/user_data=I2Nsb3VkLWNvbmZpZwpzc2hfcHdhdXRoOiB0cnVlCmRpc2FibGVfcm9vdDogMApvdXRwdXQ6CiAg >YWxsOiAnPj4gL3Zhci9sb2cvY2xvdWQtaW5pdC1vdXRwdXQubG9nJwpjaHBhc3N3ZDoKICBleHBp >cmU6IGZhbHNlCnJ1bmNtZDoKLSAnc2VkIC1pICcnL15kYXRhc291cmNlX2xpc3Q6IC9kJycgL2V0 >Yy9jbG91ZC9jbG91ZC5jZmc7IGVjaG8gJydkYXRhc291cmNlX2xpc3Q6CiAgWyJOb0Nsb3VkIiwg >IkNvbmZpZ0RyaXZlIl0nJyA+PiAvZXRjL2Nsb3VkL2Nsb3VkLmNmZycK >, openstack/content/0000=YXV0bwo= >}, volId=config-2}}, path=, device=cdrom, type=disk, readonly=true, deviceId=fc9306e3-ca33-4e96-9f08-c822bb1ee57c}, {specParams={vram=32768, heads=1}, device=qxl, type=video, deviceId=81860101-8c00-4aa1-9d32-bdc52a009f2c}, {shared=false, iface=ide, index=2, specParams={path=}, path=, device=cdrom, type=disk, readonly=true, deviceId=8e29dccb-8a92-43ab-94f4-d0ec7540dedf}, {shared=false, volumeID=860b11ae-d1a7-4c83-ad10-4989fcca0638, index=0, iface=virtio, imageID=43a334b9-1e3f-472c-b32e-e1ed0c074c14, domainID=71f82fcd-4fa6-4f56-b228-f4fa4bd4f752, specParams={}, optional=false, propagateErrors=off, device=disk, poolID=00000002-0002-0002-0002-0000000001d9, format=raw, type=disk, readonly=false, deviceId=43a334b9-1e3f-472c-b32e-e1ed0c074c14}, {nicModel=pv, specParams={outbound={}, inbound={}}, macAddr=00:1a:4a:ae:a8:01, device=bridge, linkActive=true, type=interface, filter=vdsm-no-mac-spoofing, network=ovirtmgmt, deviceId=6b19a1fd-543e-41f4-93cf-9d406414ea8e}, {index=0, model=virtio-scsi, specParams={}, device=scsi, type=controller, deviceId=1f6ab695-7327-4787-aa54-f472d3e37b60}],vmName=new-one,cpuType=Westmere,fileTransferEnable=true >2014-07-25 12:37:24,798 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVmFromCloudInitVDSCommand] (org.ovirt.thread.pool-8-thread-14) [48ba118e] FINISH, CreateVmFromCloudInitVDSCommand, log id: 598c9d44 >2014-07-25 12:37:24,809 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (org.ovirt.thread.pool-8-thread-14) [48ba118e] FINISH, CreateVmVDSCommand, return: WaitForLaunch, log id: 1498ae3e >2014-07-25 12:37:24,810 INFO [org.ovirt.engine.core.bll.RunVmCommand] (org.ovirt.thread.pool-8-thread-14) [48ba118e] Lock freed to object EngineLock [exclusiveLocks= key: 6f56ce08-46a9-4bfe-b207-50fd71e003cc value: VM >, sharedLocks= ] >2014-07-25 12:37:24,828 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-14) [48ba118e] Correlation ID: 48ba118e, Job ID: 3d6d18b7-2f82-41b0-bf6c-e4a5d6a91e70, Call Stack: null, Custom Event ID: -1, Message: VM new-one was started by admin (Host: rack). >2014-07-25 12:37:28,006 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-13) VM new-one 6f56ce08-46a9-4bfe-b207-50fd71e003cc moved from WaitForLaunch --> PoweringUp >2014-07-25 12:37:28,018 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVdsCommand] (DefaultQuartzScheduler_Worker-13) START, FullListVdsCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b], vmIds=[6f56ce08-46a9-4bfe-b207-50fd71e003cc]), log id: 720b0f07 >2014-07-25 12:37:28,034 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVdsCommand] (DefaultQuartzScheduler_Worker-13) FINISH, FullListVdsCommand, return: [{displaySecurePort=5901, kvmEnable=true, nicModel=rtl8139,pv, keyboardLayout=en-us, displayIp=10.34.62.208, pauseCode=NOERR, nice=0, displayNetwork=ovirtmgmt, copyPasteEnable=true, timeOffset=0, transparentHugePages=true, vmId=6f56ce08-46a9-4bfe-b207-50fd71e003cc, acpiEnable=true, custom={}, spiceSslCipherSuite=DEFAULT, memSize=1024, smp=1, displayPort=5900, emulatedMachine=rhel6.5.0, vmType=kvm, status=Up, memGuaranteedSize=1024, display=qxl, pid=18551, smartcardEnable=false, bootMenuEnable=false, spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,susbredir,ssmartcard, numaTune={mode=preferred}, smpCoresPerSocket=1, maxVCpus=160, clientIp=, devices=[Ljava.lang.Object;@3407cf0f, vmName=new-one, fileTransferEnable=true, cpuType=Westmere}], log id: 720b0f07 >2014-07-25 12:37:28,042 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-13) Received a memballoon Device without an address when processing VM 6f56ce08-46a9-4bfe-b207-50fd71e003cc devices, skipping device: {alias=balloon0, specParams={model=none}, device=memballoon, type=balloon} >2014-07-25 12:37:28,044 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-13) Received a spice Device without an address when processing VM 6f56ce08-46a9-4bfe-b207-50fd71e003cc devices, skipping device: {port=5900, specParams={spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,susbredir,ssmartcard, keyMap=en-us, displayNetwork=ovirtmgmt, copyPasteEnable=true, displayIp=10.34.62.208}, device=spice, tlsPort=5901, type=graphics} >2014-07-25 12:37:36,023 INFO [org.ovirt.engine.core.bll.RunVmCommand] (ajp--127.0.0.1-8702-2) [5cdd9d25] Lock Acquired to object EngineLock [exclusiveLocks= key: f7613d6f-8091-4d21-90e6-c13219a1b3db value: VM >, sharedLocks= ] >2014-07-25 12:37:36,122 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (ajp--127.0.0.1-8702-2) [5cdd9d25] START, IsVmDuringInitiatingVDSCommand( vmId = f7613d6f-8091-4d21-90e6-c13219a1b3db), log id: 2b3a75aa >2014-07-25 12:37:36,123 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (ajp--127.0.0.1-8702-2) [5cdd9d25] FINISH, IsVmDuringInitiatingVDSCommand, return: false, log id: 2b3a75aa >2014-07-25 12:37:36,220 INFO [org.ovirt.engine.core.bll.RunVmCommand] (org.ovirt.thread.pool-8-thread-31) [5cdd9d25] Running command: RunVmCommand internal: false. Entities affected : ID: f7613d6f-8091-4d21-90e6-c13219a1b3db Type: VM >2014-07-25 12:37:36,273 INFO [org.ovirt.engine.core.vdsbroker.UpdateVmDynamicDataVDSCommand] (org.ovirt.thread.pool-8-thread-31) [5cdd9d25] START, UpdateVmDynamicDataVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vmDynamic=org.ovirt.engine.core.common.businessentities.VmDynamic@f168cd40), log id: 713ee3ec >2014-07-25 12:37:36,281 INFO [org.ovirt.engine.core.vdsbroker.UpdateVmDynamicDataVDSCommand] (org.ovirt.thread.pool-8-thread-31) [5cdd9d25] FINISH, UpdateVmDynamicDataVDSCommand, log id: 713ee3ec >2014-07-25 12:37:36,293 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (org.ovirt.thread.pool-8-thread-31) [5cdd9d25] START, CreateVmVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vmId=f7613d6f-8091-4d21-90e6-c13219a1b3db, vm=VM [runit]), log id: 5a3b3c0d >2014-07-25 12:37:36,310 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVmFromCloudInitVDSCommand] (org.ovirt.thread.pool-8-thread-31) [5cdd9d25] START, CreateVmFromCloudInitVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vmId=f7613d6f-8091-4d21-90e6-c13219a1b3db, vm=VM [runit]), log id: 590a9463 >2014-07-25 12:37:36,378 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (org.ovirt.thread.pool-8-thread-31) [5cdd9d25] org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVmFromCloudInitVDSCommand kvmEnable=true,keyboardLayout=en-us,nice=0,displayNetwork=ovirtmgmt,copyPasteEnable=true,timeOffset=0,transparentHugePages=true,vmId=f7613d6f-8091-4d21-90e6-c13219a1b3db,acpiEnable=true,custom={},spiceSslCipherSuite=DEFAULT,memSize=1024,smp=1,emulatedMachine=rhel6.5.0,vmType=kvm,memGuaranteedSize=1024,display=qxl,smartcardEnable=false,bootMenuEnable=false,spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,susbredir,ssmartcard,smpCoresPerSocket=1,numaTune={mode=preferred},maxVCpus=160,devices=[{shared=false, iface=ide, index=3, specParams={vmPayload={file={openstack/latest/meta_data.json=ewogICJsYXVuY2hfaW5kZXgiIDogIjAiLAogICJhdmFpbGFiaWxpdHlfem9uZSIgOiAibm92YSIs >CiAgIm5ldHdvcmstaW50ZXJmYWNlcyIgOiAiYXV0b1xuIiwKICAibmV0d29ya19jb25maWciIDog >ewogICAgImNvbnRlbnRfcGF0aCIgOiAiL2NvbnRlbnQvMDAwMCIsCiAgICAicGF0aCIgOiAiL2V0 >Yy9uZXR3b3JrL2ludGVyZmFjZXMiCiAgfSwKICAidXVpZCIgOiAiNDAyZjY1ZTMtZTg4My00ZTE0 >LTkxNjItOTVjZjAzODg5MzY0IiwKICAibWV0YSIgOiB7CiAgICAiZXNzZW50aWFsIiA6ICJmYWxz >ZSIsCiAgICAicm9sZSIgOiAic2VydmVyIiwKICAgICJkc21vZGUiIDogImxvY2FsIgogIH0KfQ== >, openstack/latest/user_data=I2Nsb3VkLWNvbmZpZwpzc2hfcHdhdXRoOiB0cnVlCmRpc2FibGVfcm9vdDogMApvdXRwdXQ6CiAg >YWxsOiAnPj4gL3Zhci9sb2cvY2xvdWQtaW5pdC1vdXRwdXQubG9nJwpjaHBhc3N3ZDoKICBleHBp >cmU6IGZhbHNlCnJ1bmNtZDoKLSAnc2VkIC1pICcnL15kYXRhc291cmNlX2xpc3Q6IC9kJycgL2V0 >Yy9jbG91ZC9jbG91ZC5jZmc7IGVjaG8gJydkYXRhc291cmNlX2xpc3Q6CiAgWyJOb0Nsb3VkIiwg >IkNvbmZpZ0RyaXZlIl0nJyA+PiAvZXRjL2Nsb3VkL2Nsb3VkLmNmZycK >, openstack/content/0000=YXV0bwo= >}, volId=config-2}}, path=, device=cdrom, type=disk, readonly=true, deviceId=7926ee2a-7604-43c1-9ec3-86ec9408b946}, {specParams={vram=32768, heads=1}, device=qxl, type=video, deviceId=2580cdcf-a605-42d7-8773-966ee1226d36}, {shared=false, iface=ide, index=2, specParams={path=}, path=, device=cdrom, type=disk, readonly=true, deviceId=326811c5-8ddf-4c9c-be7d-7c994b7a8c19}, {shared=false, index=0, volumeID=827408c6-d52c-4c49-92f6-461ce222e485, propagateErrors=off, format=raw, type=disk, iface=virtio, bootOrder=1, domainID=71f82fcd-4fa6-4f56-b228-f4fa4bd4f752, imageID=1f7ff79b-b034-47ad-8864-181be4e1de5a, specParams={}, optional=false, device=disk, poolID=00000002-0002-0002-0002-0000000001d9, readonly=false, deviceId=1f7ff79b-b034-47ad-8864-181be4e1de5a}, {nicModel=pv, specParams={outbound={}, inbound={}}, macAddr=00:1a:4a:ae:a8:00, device=bridge, linkActive=true, type=interface, filter=vdsm-no-mac-spoofing, network=ovirtmgmt, deviceId=7341cbff-1885-4900-b72f-29066e4a550d}, {index=0, model=virtio-scsi, specParams={}, device=scsi, type=controller, deviceId=08ab4b0f-3cf3-4f84-a470-cee120c62a26}],vmName=runit,cpuType=Westmere,fileTransferEnable=true >2014-07-25 12:37:36,431 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVmFromCloudInitVDSCommand] (org.ovirt.thread.pool-8-thread-31) [5cdd9d25] FINISH, CreateVmFromCloudInitVDSCommand, log id: 590a9463 >2014-07-25 12:37:36,438 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (org.ovirt.thread.pool-8-thread-31) [5cdd9d25] FINISH, CreateVmVDSCommand, return: WaitForLaunch, log id: 5a3b3c0d >2014-07-25 12:37:36,439 INFO [org.ovirt.engine.core.bll.RunVmCommand] (org.ovirt.thread.pool-8-thread-31) [5cdd9d25] Lock freed to object EngineLock [exclusiveLocks= key: f7613d6f-8091-4d21-90e6-c13219a1b3db value: VM >, sharedLocks= ] >2014-07-25 12:37:36,452 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-31) [5cdd9d25] Correlation ID: 5cdd9d25, Job ID: 98a33085-e41a-4c6a-830b-138aaa35f2e6, Call Stack: null, Custom Event ID: -1, Message: VM runit was started by admin (Host: rack). >2014-07-25 12:37:37,253 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVdsCommand] (DefaultQuartzScheduler_Worker-33) START, FullListVdsCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b], vmIds=[f7613d6f-8091-4d21-90e6-c13219a1b3db]), log id: 5696275e >2014-07-25 12:37:37,283 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVdsCommand] (DefaultQuartzScheduler_Worker-33) FINISH, FullListVdsCommand, return: [{displaySecurePort=5903, kvmEnable=true, nicModel=rtl8139,pv, keyboardLayout=en-us, displayIp=10.34.62.208, pauseCode=NOERR, nice=0, displayNetwork=ovirtmgmt, copyPasteEnable=true, timeOffset=0, transparentHugePages=true, vmId=f7613d6f-8091-4d21-90e6-c13219a1b3db, acpiEnable=true, custom={}, spiceSslCipherSuite=DEFAULT, memSize=1024, smp=1, displayPort=5902, emulatedMachine=rhel6.5.0, vmType=kvm, status=Up, memGuaranteedSize=1024, display=qxl, pid=18709, smartcardEnable=false, bootMenuEnable=false, spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,susbredir,ssmartcard, numaTune={mode=preferred}, smpCoresPerSocket=1, maxVCpus=160, clientIp=, devices=[Ljava.lang.Object;@5c3e0b52, vmName=runit, fileTransferEnable=true, cpuType=Westmere}], log id: 5696275e >2014-07-25 12:37:37,294 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-33) Received a memballoon Device without an address when processing VM f7613d6f-8091-4d21-90e6-c13219a1b3db devices, skipping device: {alias=balloon0, specParams={model=none}, device=memballoon, type=balloon} >2014-07-25 12:37:37,296 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-33) Received a spice Device without an address when processing VM f7613d6f-8091-4d21-90e6-c13219a1b3db devices, skipping device: {port=5902, specParams={spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,susbredir,ssmartcard, keyMap=en-us, displayNetwork=ovirtmgmt, copyPasteEnable=true, displayIp=10.34.62.208}, device=spice, tlsPort=5903, type=graphics} >2014-07-25 12:37:40,388 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-40) VM runit f7613d6f-8091-4d21-90e6-c13219a1b3db moved from WaitForLaunch --> PoweringUp >2014-07-25 12:37:54,233 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (DefaultQuartzScheduler_Worker-67) Setting new tasks map. The map contains now 1 tasks >2014-07-25 12:37:58,127 INFO [org.ovirt.engine.core.bll.MultipleActionsRunner] (org.ovirt.thread.pool-8-thread-21) [5eb231f3] Start running CanDoAction for command number 1/2 (Command type: StopVm) >2014-07-25 12:37:58,127 INFO [org.ovirt.engine.core.bll.MultipleActionsRunner] (org.ovirt.thread.pool-8-thread-19) [4ed7917] Start running CanDoAction for command number 2/2 (Command type: StopVm) >2014-07-25 12:37:58,154 INFO [org.ovirt.engine.core.bll.MultipleActionsRunner] (org.ovirt.thread.pool-8-thread-19) [4ed7917] Finish handling CanDoAction for command number 2/2 (Command type: StopVm) >2014-07-25 12:37:58,164 INFO [org.ovirt.engine.core.bll.MultipleActionsRunner] (org.ovirt.thread.pool-8-thread-21) [5eb231f3] Finish handling CanDoAction for command number 1/2 (Command type: StopVm) >2014-07-25 12:37:58,190 INFO [org.ovirt.engine.core.bll.StopVmCommand] (org.ovirt.thread.pool-8-thread-17) [5eb231f3] Running command: StopVmCommand internal: false. Entities affected : ID: 6f56ce08-46a9-4bfe-b207-50fd71e003cc Type: VM >2014-07-25 12:37:58,200 INFO [org.ovirt.engine.core.vdsbroker.DestroyVmVDSCommand] (org.ovirt.thread.pool-8-thread-17) [5eb231f3] START, DestroyVmVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vmId=6f56ce08-46a9-4bfe-b207-50fd71e003cc, force=false, secondsToWait=0, gracefully=false, reason=), log id: 328ee15c >2014-07-25 12:37:58,224 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (org.ovirt.thread.pool-8-thread-17) [5eb231f3] START, DestroyVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vmId=6f56ce08-46a9-4bfe-b207-50fd71e003cc, force=false, secondsToWait=0, gracefully=false, reason=), log id: 2e156077 >2014-07-25 12:37:58,642 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (org.ovirt.thread.pool-8-thread-17) [5eb231f3] FINISH, DestroyVDSCommand, log id: 2e156077 >2014-07-25 12:37:58,652 INFO [org.ovirt.engine.core.vdsbroker.DestroyVmVDSCommand] (org.ovirt.thread.pool-8-thread-17) [5eb231f3] FINISH, DestroyVmVDSCommand, return: Down, log id: 328ee15c >2014-07-25 12:37:58,659 INFO [org.ovirt.engine.core.bll.ProcessDownVmCommand] (org.ovirt.thread.pool-8-thread-22) [259e5d4c] Running command: ProcessDownVmCommand internal: true. >2014-07-25 12:37:58,667 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-17) [5eb231f3] Correlation ID: 5eb231f3, Job ID: 5f88f5c8-0263-4e61-bfb2-97fb00d16aa4, Call Stack: null, Custom Event ID: -1, Message: VM new-one powered off by admin (Host: rack) (Reason: Not Specified). >2014-07-25 12:37:58,694 INFO [org.ovirt.engine.core.bll.StopVmCommand] (org.ovirt.thread.pool-8-thread-17) [4ed7917] Running command: StopVmCommand internal: false. Entities affected : ID: f7613d6f-8091-4d21-90e6-c13219a1b3db Type: VM >2014-07-25 12:37:58,699 INFO [org.ovirt.engine.core.vdsbroker.DestroyVmVDSCommand] (org.ovirt.thread.pool-8-thread-17) [4ed7917] START, DestroyVmVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vmId=f7613d6f-8091-4d21-90e6-c13219a1b3db, force=false, secondsToWait=0, gracefully=false, reason=), log id: 763db38b >2014-07-25 12:37:58,711 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (org.ovirt.thread.pool-8-thread-17) [4ed7917] START, DestroyVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vmId=f7613d6f-8091-4d21-90e6-c13219a1b3db, force=false, secondsToWait=0, gracefully=false, reason=), log id: 500c5c1a >2014-07-25 12:37:59,123 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (org.ovirt.thread.pool-8-thread-17) [4ed7917] FINISH, DestroyVDSCommand, log id: 500c5c1a >2014-07-25 12:37:59,132 INFO [org.ovirt.engine.core.vdsbroker.DestroyVmVDSCommand] (org.ovirt.thread.pool-8-thread-17) [4ed7917] FINISH, DestroyVmVDSCommand, return: Down, log id: 763db38b >2014-07-25 12:37:59,143 INFO [org.ovirt.engine.core.bll.ProcessDownVmCommand] (org.ovirt.thread.pool-8-thread-24) [d6d8943] Running command: ProcessDownVmCommand internal: true. >2014-07-25 12:37:59,144 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-17) [4ed7917] Correlation ID: 4ed7917, Job ID: 6fca0289-fce3-4613-ac1a-b6feaf49ad8f, Call Stack: null, Custom Event ID: -1, Message: VM runit powered off by admin (Host: rack) (Reason: Not Specified). >2014-07-25 12:38:24,235 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (DefaultQuartzScheduler_Worker-35) Setting new tasks map. The map contains now 0 tasks >2014-07-25 12:38:24,236 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (DefaultQuartzScheduler_Worker-35) Cleared all tasks of pool 00000002-0002-0002-0002-0000000001d9. >2014-07-25 12:39:01,909 WARN [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-8-thread-47) Executing a command: java.util.concurrent.FutureTask , but note that there are 1 tasks in the queue. >2014-07-25 12:39:01,910 INFO [org.ovirt.engine.core.bll.MultipleActionsRunner] (org.ovirt.thread.pool-8-thread-47) [25552b90] Start running CanDoAction for command number 1/2 (Command type: RemoveVm) >2014-07-25 12:39:01,911 INFO [org.ovirt.engine.core.bll.MultipleActionsRunner] (org.ovirt.thread.pool-8-thread-48) [79983e87] Start running CanDoAction for command number 2/2 (Command type: RemoveVm) >2014-07-25 12:39:01,916 INFO [org.ovirt.engine.core.bll.RemoveVmCommand] (org.ovirt.thread.pool-8-thread-48) [79983e87] Lock Acquired to object EngineLock [exclusiveLocks= key: f7613d6f-8091-4d21-90e6-c13219a1b3db value: VM >, sharedLocks= ] >2014-07-25 12:39:01,917 INFO [org.ovirt.engine.core.bll.RemoveVmCommand] (org.ovirt.thread.pool-8-thread-47) [25552b90] Lock Acquired to object EngineLock [exclusiveLocks= key: 6f56ce08-46a9-4bfe-b207-50fd71e003cc value: VM >, sharedLocks= ] >2014-07-25 12:39:01,951 INFO [org.ovirt.engine.core.bll.MultipleActionsRunner] (org.ovirt.thread.pool-8-thread-47) [25552b90] Finish handling CanDoAction for command number 1/2 (Command type: RemoveVm) >2014-07-25 12:39:01,974 INFO [org.ovirt.engine.core.bll.MultipleActionsRunner] (org.ovirt.thread.pool-8-thread-48) [79983e87] Finish handling CanDoAction for command number 2/2 (Command type: RemoveVm) >2014-07-25 12:39:01,998 INFO [org.ovirt.engine.core.bll.RemoveVmCommand] (org.ovirt.thread.pool-8-thread-49) [25552b90] Running command: RemoveVmCommand internal: false. Entities affected : ID: 6f56ce08-46a9-4bfe-b207-50fd71e003cc Type: VM >2014-07-25 12:39:01,999 INFO [org.ovirt.engine.core.vdsbroker.SetVmStatusVDSCommand] (org.ovirt.thread.pool-8-thread-49) [25552b90] START, SetVmStatusVDSCommand( vmId = 6f56ce08-46a9-4bfe-b207-50fd71e003cc, status = ImageLocked, exit status = Normal), log id: 3792fa71 >2014-07-25 12:39:02,006 INFO [org.ovirt.engine.core.vdsbroker.SetVmStatusVDSCommand] (org.ovirt.thread.pool-8-thread-49) [25552b90] FINISH, SetVmStatusVDSCommand, log id: 3792fa71 >2014-07-25 12:39:02,010 INFO [org.ovirt.engine.core.bll.RemoveVmCommand] (org.ovirt.thread.pool-8-thread-49) [25552b90] Lock freed to object EngineLock [exclusiveLocks= key: 6f56ce08-46a9-4bfe-b207-50fd71e003cc value: VM >, sharedLocks= ] >2014-07-25 12:39:02,059 INFO [org.ovirt.engine.core.bll.RemoveAllVmImagesCommand] (org.ovirt.thread.pool-8-thread-49) [60eabe5a] Running command: RemoveAllVmImagesCommand internal: true. Entities affected : ID: 6f56ce08-46a9-4bfe-b207-50fd71e003cc Type: VM >2014-07-25 12:39:02,072 INFO [org.ovirt.engine.core.bll.RemoveImageCommand] (org.ovirt.thread.pool-8-thread-49) [2de72aba] Running command: RemoveImageCommand internal: true. Entities affected : ID: 00000000-0000-0000-0000-000000000000 Type: Storage >2014-07-25 12:39:02,171 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.DeleteImageGroupVDSCommand] (org.ovirt.thread.pool-8-thread-49) [2de72aba] START, DeleteImageGroupVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, storageDomainId = 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752, imageGroupId = 43a334b9-1e3f-472c-b32e-e1ed0c074c14, postZeros = false, forceDelete = false), log id: 149bdd1d >2014-07-25 12:39:10,693 INFO [org.ovirt.engine.core.bll.RemoveVmTemplateCommand] (ajp--127.0.0.1-8702-2) [7eafb87d] Lock Acquired to object EngineLock [exclusiveLocks= key: d031b0a2-efae-4300-b703-af169651be32 value: TEMPLATE >, sharedLocks= ] >2014-07-25 12:39:10,785 INFO [org.ovirt.engine.core.bll.RemoveVmTemplateCommand] (org.ovirt.thread.pool-8-thread-11) [7eafb87d] Running command: RemoveVmTemplateCommand internal: false. Entities affected : ID: d031b0a2-efae-4300-b703-af169651be32 Type: VmTemplate >2014-07-25 12:39:10,806 INFO [org.ovirt.engine.core.bll.RemoveAllVmTemplateImageTemplatesCommand] (org.ovirt.thread.pool-8-thread-11) [7eafb87d] Running command: RemoveAllVmTemplateImageTemplatesCommand internal: true. Entities affected : ID: d031b0a2-efae-4300-b703-af169651be32 Type: VmTemplate >2014-07-25 12:39:10,827 INFO [org.ovirt.engine.core.bll.RemoveTemplateSnapshotCommand] (org.ovirt.thread.pool-8-thread-11) [7a73c920] Running command: RemoveTemplateSnapshotCommand internal: true. Entities affected : ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage >2014-07-25 12:39:10,859 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.DeleteImageGroupVDSCommand] (org.ovirt.thread.pool-8-thread-11) [7a73c920] START, DeleteImageGroupVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, storageDomainId = 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752, imageGroupId = 4ace1a06-1fa4-4d9f-8442-fc3e2a3def19, postZeros = false, forceDelete = false), log id: 68836761 >2014-07-25 12:39:12,318 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.DeleteImageGroupVDSCommand] (org.ovirt.thread.pool-8-thread-49) [2de72aba] FINISH, DeleteImageGroupVDSCommand, log id: 149bdd1d >2014-07-25 12:39:12,440 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-8-thread-49) [2de72aba] CommandAsyncTask::Adding CommandMultiAsyncTasks object for command 199afc57-2e3a-4827-9da2-48aaf3f9cb5a >2014-07-25 12:39:12,441 INFO [org.ovirt.engine.core.bll.CommandMultiAsyncTasks] (org.ovirt.thread.pool-8-thread-49) [2de72aba] CommandMultiAsyncTasks::AttachTask: Attaching task db8a1b94-2090-4c17-b6ba-f686f03e2b01 to command 199afc57-2e3a-4827-9da2-48aaf3f9cb5a. >2014-07-25 12:39:12,453 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (org.ovirt.thread.pool-8-thread-49) [2de72aba] Adding task db8a1b94-2090-4c17-b6ba-f686f03e2b01 (Parent Command RemoveVm, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters), polling hasn't started yet.. >2014-07-25 12:39:12,528 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-49) [2de72aba] Correlation ID: 25552b90, Job ID: 785d6e4c-b858-4f0f-b3d0-f38642c23434, Call Stack: null, Custom Event ID: -1, Message: VM new-one was successfully removed. >2014-07-25 12:39:12,529 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (org.ovirt.thread.pool-8-thread-49) [2de72aba] BaseAsyncTask::startPollingTask: Starting to poll task db8a1b94-2090-4c17-b6ba-f686f03e2b01. >2014-07-25 12:39:12,543 INFO [org.ovirt.engine.core.bll.RemoveVmCommand] (org.ovirt.thread.pool-8-thread-49) [79983e87] Running command: RemoveVmCommand internal: false. Entities affected : ID: f7613d6f-8091-4d21-90e6-c13219a1b3db Type: VM >2014-07-25 12:39:12,544 INFO [org.ovirt.engine.core.vdsbroker.SetVmStatusVDSCommand] (org.ovirt.thread.pool-8-thread-49) [79983e87] START, SetVmStatusVDSCommand( vmId = f7613d6f-8091-4d21-90e6-c13219a1b3db, status = ImageLocked, exit status = Normal), log id: c196c0c >2014-07-25 12:39:12,546 INFO [org.ovirt.engine.core.vdsbroker.SetVmStatusVDSCommand] (org.ovirt.thread.pool-8-thread-49) [79983e87] FINISH, SetVmStatusVDSCommand, log id: c196c0c >2014-07-25 12:39:12,549 INFO [org.ovirt.engine.core.bll.RemoveVmCommand] (org.ovirt.thread.pool-8-thread-49) [79983e87] Lock freed to object EngineLock [exclusiveLocks= key: f7613d6f-8091-4d21-90e6-c13219a1b3db value: VM >, sharedLocks= ] >2014-07-25 12:39:12,558 INFO [org.ovirt.engine.core.bll.RemoveAllVmImagesCommand] (org.ovirt.thread.pool-8-thread-49) [1fd9302b] Running command: RemoveAllVmImagesCommand internal: true. Entities affected : ID: f7613d6f-8091-4d21-90e6-c13219a1b3db Type: VM >2014-07-25 12:39:12,561 INFO [org.ovirt.engine.core.bll.RemoveImageCommand] (org.ovirt.thread.pool-8-thread-49) [5a453713] Running command: RemoveImageCommand internal: true. Entities affected : ID: 00000000-0000-0000-0000-000000000000 Type: Storage >2014-07-25 12:39:12,567 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.DeleteImageGroupVDSCommand] (org.ovirt.thread.pool-8-thread-49) [5a453713] START, DeleteImageGroupVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, storageDomainId = 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752, imageGroupId = 1f7ff79b-b034-47ad-8864-181be4e1de5a, postZeros = false, forceDelete = false), log id: 18912c0e >2014-07-25 12:39:14,411 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (DefaultQuartzScheduler_Worker-50) Polling and updating Async Tasks: 1 tasks, 1 tasks to poll now >2014-07-25 12:39:14,968 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.DeleteImageGroupVDSCommand] (org.ovirt.thread.pool-8-thread-11) [7a73c920] FINISH, DeleteImageGroupVDSCommand, log id: 68836761 >2014-07-25 12:39:14,993 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-8-thread-11) [7a73c920] CommandAsyncTask::Adding CommandMultiAsyncTasks object for command e646e1a1-1d54-48ac-801e-b5ab11ea7c27 >2014-07-25 12:39:14,994 INFO [org.ovirt.engine.core.bll.CommandMultiAsyncTasks] (org.ovirt.thread.pool-8-thread-11) [7a73c920] CommandMultiAsyncTasks::AttachTask: Attaching task 3ede13e8-c87d-4709-adef-0e890e382020 to command e646e1a1-1d54-48ac-801e-b5ab11ea7c27. >2014-07-25 12:39:15,053 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (DefaultQuartzScheduler_Worker-50) SPMAsyncTask::PollTask: Polling task db8a1b94-2090-4c17-b6ba-f686f03e2b01 (Parent Command RemoveVm, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) returned status finished, result 'success'. >2014-07-25 12:39:15,059 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (DefaultQuartzScheduler_Worker-50) BaseAsyncTask::OnTaskEndSuccess: Task db8a1b94-2090-4c17-b6ba-f686f03e2b01 (Parent Command RemoveVm, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) ended successfully. >2014-07-25 12:39:15,061 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (DefaultQuartzScheduler_Worker-50) CommandAsyncTask::EndActionIfNecessary: All tasks of command 199afc57-2e3a-4827-9da2-48aaf3f9cb5a has ended -> executing endAction >2014-07-25 12:39:15,062 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (DefaultQuartzScheduler_Worker-50) CommandAsyncTask::endAction: Ending action for 1 tasks (command ID: 199afc57-2e3a-4827-9da2-48aaf3f9cb5a): calling endAction . >2014-07-25 12:39:15,063 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (org.ovirt.thread.pool-8-thread-11) [7a73c920] Adding task 3ede13e8-c87d-4709-adef-0e890e382020 (Parent Command RemoveVmTemplate, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters), polling hasn't started yet.. >2014-07-25 12:39:15,063 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-8-thread-15) CommandAsyncTask::EndCommandAction [within thread] context: Attempting to endAction RemoveVm, executionIndex: 0 >2014-07-25 12:39:15,086 INFO [org.ovirt.engine.core.bll.RemoveVmCommand] (org.ovirt.thread.pool-8-thread-15) Ending command successfully: org.ovirt.engine.core.bll.RemoveVmCommand >2014-07-25 12:39:15,088 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-8-thread-15) CommandAsyncTask::HandleEndActionResult [within thread]: endAction for action type RemoveVm completed, handling the result. >2014-07-25 12:39:15,090 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-8-thread-15) CommandAsyncTask::HandleEndActionResult [within thread]: endAction for action type RemoveVm succeeded, clearing tasks. >2014-07-25 12:39:15,095 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (org.ovirt.thread.pool-8-thread-15) SPMAsyncTask::ClearAsyncTask: Attempting to clear task db8a1b94-2090-4c17-b6ba-f686f03e2b01 >2014-07-25 12:39:15,096 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-15) START, SPMClearTaskVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, taskId = db8a1b94-2090-4c17-b6ba-f686f03e2b01), log id: 65ec8115 >2014-07-25 12:39:15,116 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-11) [7a73c920] Correlation ID: 7eafb87d, Job ID: ab47d22c-85f3-4539-9e4b-ecba3c57dcc1, Call Stack: null, Custom Event ID: -1, Message: Removal of Template blank-disk was initiated by admin. >2014-07-25 12:39:15,117 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (org.ovirt.thread.pool-8-thread-11) [7a73c920] BaseAsyncTask::startPollingTask: Starting to poll task 3ede13e8-c87d-4709-adef-0e890e382020. >2014-07-25 12:39:25,067 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (DefaultQuartzScheduler_Worker-77) Polling and updating Async Tasks: 2 tasks, 1 tasks to poll now >2014-07-25 12:39:28,534 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.DeleteImageGroupVDSCommand] (org.ovirt.thread.pool-8-thread-49) [5a453713] FINISH, DeleteImageGroupVDSCommand, log id: 18912c0e >2014-07-25 12:39:28,540 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (DefaultQuartzScheduler_Worker-77) SPMAsyncTask::PollTask: Polling task 3ede13e8-c87d-4709-adef-0e890e382020 (Parent Command RemoveVmTemplate, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) returned status finished, result 'success'. >2014-07-25 12:39:28,548 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-8-thread-49) [5a453713] CommandAsyncTask::Adding CommandMultiAsyncTasks object for command f1e78c90-a974-4bef-a294-edc2d7b21549 >2014-07-25 12:39:28,549 INFO [org.ovirt.engine.core.bll.CommandMultiAsyncTasks] (org.ovirt.thread.pool-8-thread-49) [5a453713] CommandMultiAsyncTasks::AttachTask: Attaching task c863b58a-c89b-4663-b2aa-2af3a97ad2c2 to command f1e78c90-a974-4bef-a294-edc2d7b21549. >2014-07-25 12:39:28,551 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (DefaultQuartzScheduler_Worker-77) BaseAsyncTask::OnTaskEndSuccess: Task 3ede13e8-c87d-4709-adef-0e890e382020 (Parent Command RemoveVmTemplate, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) ended successfully. >2014-07-25 12:39:28,563 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (DefaultQuartzScheduler_Worker-77) CommandAsyncTask::EndActionIfNecessary: All tasks of command e646e1a1-1d54-48ac-801e-b5ab11ea7c27 has ended -> executing endAction >2014-07-25 12:39:28,565 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (DefaultQuartzScheduler_Worker-77) CommandAsyncTask::endAction: Ending action for 1 tasks (command ID: e646e1a1-1d54-48ac-801e-b5ab11ea7c27): calling endAction . >2014-07-25 12:39:28,566 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-8-thread-29) CommandAsyncTask::EndCommandAction [within thread] context: Attempting to endAction RemoveVmTemplate, executionIndex: 0 >2014-07-25 12:39:28,586 INFO [org.ovirt.engine.core.bll.RemoveVmTemplateCommand] (org.ovirt.thread.pool-8-thread-29) Ending command successfully: org.ovirt.engine.core.bll.RemoveVmTemplateCommand >2014-07-25 12:39:28,589 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (org.ovirt.thread.pool-8-thread-49) [5a453713] Adding task c863b58a-c89b-4663-b2aa-2af3a97ad2c2 (Parent Command RemoveVm, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters), polling hasn't started yet.. >2014-07-25 12:39:28,605 INFO [org.ovirt.engine.core.bll.RemoveVmTemplateCommand] (org.ovirt.thread.pool-8-thread-29) Lock freed to object EngineLock [exclusiveLocks= key: d031b0a2-efae-4300-b703-af169651be32 value: TEMPLATE >, sharedLocks= ] >2014-07-25 12:39:28,629 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-29) Correlation ID: 7eafb87d, Call Stack: null, Custom Event ID: -1, Message: Removal of Template blank-disk has been completed. >2014-07-25 12:39:28,631 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-8-thread-29) CommandAsyncTask::HandleEndActionResult [within thread]: endAction for action type RemoveVmTemplate completed, handling the result. >2014-07-25 12:39:28,632 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-8-thread-29) CommandAsyncTask::HandleEndActionResult [within thread]: endAction for action type RemoveVmTemplate succeeded, clearing tasks. >2014-07-25 12:39:28,637 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-15) START, HSMClearTaskVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, taskId=db8a1b94-2090-4c17-b6ba-f686f03e2b01), log id: 1666fcb0 >2014-07-25 12:39:28,641 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (org.ovirt.thread.pool-8-thread-29) SPMAsyncTask::ClearAsyncTask: Attempting to clear task 3ede13e8-c87d-4709-adef-0e890e382020 >2014-07-25 12:39:28,642 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-29) START, SPMClearTaskVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, taskId = 3ede13e8-c87d-4709-adef-0e890e382020), log id: 11b62e66 >2014-07-25 12:39:28,706 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-49) [5a453713] Correlation ID: 79983e87, Job ID: 29489b48-6f2a-444b-90fe-0f2995907c5d, Call Stack: null, Custom Event ID: -1, Message: VM runit was successfully removed. >2014-07-25 12:39:28,707 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (org.ovirt.thread.pool-8-thread-49) [5a453713] BaseAsyncTask::startPollingTask: Starting to poll task c863b58a-c89b-4663-b2aa-2af3a97ad2c2. >2014-07-25 12:39:37,511 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-15) FINISH, HSMClearTaskVDSCommand, log id: 1666fcb0 >2014-07-25 12:39:37,512 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-15) FINISH, SPMClearTaskVDSCommand, log id: 65ec8115 >2014-07-25 12:39:37,514 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-29) START, HSMClearTaskVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, taskId=3ede13e8-c87d-4709-adef-0e890e382020), log id: 1767915c >2014-07-25 12:39:37,523 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (org.ovirt.thread.pool-8-thread-15) BaseAsyncTask::RemoveTaskFromDB: Removed task db8a1b94-2090-4c17-b6ba-f686f03e2b01 from DataBase >2014-07-25 12:39:37,524 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-8-thread-15) CommandAsyncTask::HandleEndActionResult [within thread]: Removing CommandMultiAsyncTasks object for entity 199afc57-2e3a-4827-9da2-48aaf3f9cb5a >2014-07-25 12:39:37,643 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-29) FINISH, HSMClearTaskVDSCommand, log id: 1767915c >2014-07-25 12:39:37,644 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-29) FINISH, SPMClearTaskVDSCommand, log id: 11b62e66 >2014-07-25 12:39:37,653 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (org.ovirt.thread.pool-8-thread-29) BaseAsyncTask::RemoveTaskFromDB: Removed task 3ede13e8-c87d-4709-adef-0e890e382020 from DataBase >2014-07-25 12:39:37,654 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-8-thread-29) CommandAsyncTask::HandleEndActionResult [within thread]: Removing CommandMultiAsyncTasks object for entity e646e1a1-1d54-48ac-801e-b5ab11ea7c27 >2014-07-25 12:39:38,567 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (DefaultQuartzScheduler_Worker-3) Polling and updating Async Tasks: 3 tasks, 1 tasks to poll now >2014-07-25 12:39:38,575 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (DefaultQuartzScheduler_Worker-3) SPMAsyncTask::PollTask: Polling task c863b58a-c89b-4663-b2aa-2af3a97ad2c2 (Parent Command RemoveVm, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) returned status finished, result 'success'. >2014-07-25 12:39:38,581 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (DefaultQuartzScheduler_Worker-3) BaseAsyncTask::OnTaskEndSuccess: Task c863b58a-c89b-4663-b2aa-2af3a97ad2c2 (Parent Command RemoveVm, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) ended successfully. >2014-07-25 12:39:38,583 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (DefaultQuartzScheduler_Worker-3) CommandAsyncTask::EndActionIfNecessary: All tasks of command f1e78c90-a974-4bef-a294-edc2d7b21549 has ended -> executing endAction >2014-07-25 12:39:38,584 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (DefaultQuartzScheduler_Worker-3) CommandAsyncTask::endAction: Ending action for 1 tasks (command ID: f1e78c90-a974-4bef-a294-edc2d7b21549): calling endAction . >2014-07-25 12:39:38,585 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-8-thread-46) CommandAsyncTask::EndCommandAction [within thread] context: Attempting to endAction RemoveVm, executionIndex: 0 >2014-07-25 12:39:38,598 INFO [org.ovirt.engine.core.bll.RemoveVmCommand] (org.ovirt.thread.pool-8-thread-46) Ending command successfully: org.ovirt.engine.core.bll.RemoveVmCommand >2014-07-25 12:39:38,601 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-8-thread-46) CommandAsyncTask::HandleEndActionResult [within thread]: endAction for action type RemoveVm completed, handling the result. >2014-07-25 12:39:38,602 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-8-thread-46) CommandAsyncTask::HandleEndActionResult [within thread]: endAction for action type RemoveVm succeeded, clearing tasks. >2014-07-25 12:39:38,607 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (org.ovirt.thread.pool-8-thread-46) SPMAsyncTask::ClearAsyncTask: Attempting to clear task c863b58a-c89b-4663-b2aa-2af3a97ad2c2 >2014-07-25 12:39:38,608 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-46) START, SPMClearTaskVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, taskId = c863b58a-c89b-4663-b2aa-2af3a97ad2c2), log id: 199864ee >2014-07-25 12:39:38,611 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-46) START, HSMClearTaskVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, taskId=c863b58a-c89b-4663-b2aa-2af3a97ad2c2), log id: 640a5262 >2014-07-25 12:39:38,697 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-46) FINISH, HSMClearTaskVDSCommand, log id: 640a5262 >2014-07-25 12:39:38,698 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-46) FINISH, SPMClearTaskVDSCommand, log id: 199864ee >2014-07-25 12:39:38,713 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (org.ovirt.thread.pool-8-thread-46) BaseAsyncTask::RemoveTaskFromDB: Removed task c863b58a-c89b-4663-b2aa-2af3a97ad2c2 from DataBase >2014-07-25 12:39:38,714 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-8-thread-46) CommandAsyncTask::HandleEndActionResult [within thread]: Removing CommandMultiAsyncTasks object for entity f1e78c90-a974-4bef-a294-edc2d7b21549 >2014-07-25 12:40:24,237 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (DefaultQuartzScheduler_Worker-82) Setting new tasks map. The map contains now 2 tasks >2014-07-25 12:40:54,238 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (DefaultQuartzScheduler_Worker-75) Setting new tasks map. The map contains now 0 tasks >2014-07-25 12:40:54,239 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (DefaultQuartzScheduler_Worker-75) Cleared all tasks of pool 00000002-0002-0002-0002-0000000001d9. >2014-07-25 13:21:50,014 INFO [org.ovirt.engine.core.bll.LoginAdminUserCommand] (ajp--127.0.0.1-8702-9) Running command: LoginAdminUserCommand internal: false. >2014-07-25 13:21:50,020 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-9) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-07-25 13:21:50,119 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp--127.0.0.1-8702-4) Running command: LoginUserCommand internal: false. >2014-07-25 13:23:54,281 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-82) Principal admin::internal synchronized >2014-07-25 13:23:54,579 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-13) Attempting to update VMs/Templates Ovf. >2014-07-25 13:23:54,814 INFO [org.ovirt.engine.core.bll.ProcessOvfUpdateForStorageDomainCommand] (DefaultQuartzScheduler_Worker-13) [18e37f83] Lock Acquired to object EngineLock [exclusiveLocks= key: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 value: STORAGE >, sharedLocks= key: 00000002-0002-0002-0002-0000000001d9 value: OVF_UPDATE >] >2014-07-25 13:23:54,824 INFO [org.ovirt.engine.core.bll.ProcessOvfUpdateForStorageDomainCommand] (DefaultQuartzScheduler_Worker-13) [18e37f83] Running command: ProcessOvfUpdateForStorageDomainCommand internal: true. Entities affected : ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage >2014-07-25 13:23:54,844 INFO [org.ovirt.engine.core.bll.CreateOvfStoresForStorageDomainCommand] (DefaultQuartzScheduler_Worker-13) [4168ff2b] Running command: CreateOvfStoresForStorageDomainCommand internal: true. >2014-07-25 13:23:54,848 INFO [org.ovirt.engine.core.bll.CreateOvfVolumeForStorageDomainCommand] (DefaultQuartzScheduler_Worker-13) [4e0946ad] Running command: CreateOvfVolumeForStorageDomainCommand internal: true. Entities affected : ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage >2014-07-25 13:23:54,865 INFO [org.ovirt.engine.core.bll.AddDiskCommand] (DefaultQuartzScheduler_Worker-13) [af148f1] Running command: AddDiskCommand internal: true. Entities affected : ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage >2014-07-25 13:23:54,887 INFO [org.ovirt.engine.core.bll.AddImageFromScratchCommand] (DefaultQuartzScheduler_Worker-13) [63bc3d2f] Running command: AddImageFromScratchCommand internal: true. Entities affected : ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage >2014-07-25 13:23:54,898 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.CreateImageVDSCommand] (DefaultQuartzScheduler_Worker-13) [63bc3d2f] START, CreateImageVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, storageDomainId = 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752, imageGroupId = 07c061c2-77f4-4c02-8402-e4dff773b216, imageSizeInBytes = 134217728, volumeFormat = RAW, newImageId = 1f5f82d1-85c2-4386-8764-02bab6dcbc21, newImageDescription = OVF_STORE), log id: 7cdb2fb7 >2014-07-25 13:23:54,904 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.CreateImageVDSCommand] (DefaultQuartzScheduler_Worker-13) [63bc3d2f] -- executeIrsBrokerCommand: calling 'createVolume' with two new parameters: description and UUID >2014-07-25 13:23:55,258 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.CreateImageVDSCommand] (DefaultQuartzScheduler_Worker-13) [63bc3d2f] FINISH, CreateImageVDSCommand, return: 1f5f82d1-85c2-4386-8764-02bab6dcbc21, log id: 7cdb2fb7 >2014-07-25 13:23:55,262 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (DefaultQuartzScheduler_Worker-13) [63bc3d2f] CommandAsyncTask::Adding CommandMultiAsyncTasks object for command 8c80d80c-39bb-4109-a465-c2cb1c952b15 >2014-07-25 13:23:55,263 INFO [org.ovirt.engine.core.bll.CommandMultiAsyncTasks] (DefaultQuartzScheduler_Worker-13) [63bc3d2f] CommandMultiAsyncTasks::AttachTask: Attaching task fc24cd46-4312-49e7-adda-5903f75e3e77 to command 8c80d80c-39bb-4109-a465-c2cb1c952b15. >2014-07-25 13:23:55,271 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (DefaultQuartzScheduler_Worker-13) [63bc3d2f] Adding task fc24cd46-4312-49e7-adda-5903f75e3e77 (Parent Command CreateOvfStoresForStorageDomain, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters), polling hasn't started yet.. >2014-07-25 13:23:55,279 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-13) [63bc3d2f] Correlation ID: af148f1, Call Stack: null, Custom Event ID: -1, Message: Add-Disk operation of 'OVF_STORE' was initiated by Non interactive user. >2014-07-25 13:23:55,292 INFO [org.ovirt.engine.core.bll.CreateOvfVolumeForStorageDomainCommand] (DefaultQuartzScheduler_Worker-13) [59d5d808] Running command: CreateOvfVolumeForStorageDomainCommand internal: true. Entities affected : ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage >2014-07-25 13:23:55,303 INFO [org.ovirt.engine.core.bll.AddDiskCommand] (DefaultQuartzScheduler_Worker-13) [665a234b] Running command: AddDiskCommand internal: true. Entities affected : ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage >2014-07-25 13:23:55,310 INFO [org.ovirt.engine.core.bll.AddImageFromScratchCommand] (DefaultQuartzScheduler_Worker-13) [4dca0641] Running command: AddImageFromScratchCommand internal: true. Entities affected : ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage >2014-07-25 13:23:55,322 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.CreateImageVDSCommand] (DefaultQuartzScheduler_Worker-13) [4dca0641] START, CreateImageVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, storageDomainId = 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752, imageGroupId = 90108fe1-56a4-4f0e-b301-fa0ef46ba717, imageSizeInBytes = 134217728, volumeFormat = RAW, newImageId = 8596469e-37df-4d9e-9793-d238238131f5, newImageDescription = OVF_STORE), log id: 3021149c >2014-07-25 13:23:55,325 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.CreateImageVDSCommand] (DefaultQuartzScheduler_Worker-13) [4dca0641] -- executeIrsBrokerCommand: calling 'createVolume' with two new parameters: description and UUID >2014-07-25 13:23:55,751 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.CreateImageVDSCommand] (DefaultQuartzScheduler_Worker-13) [4dca0641] FINISH, CreateImageVDSCommand, return: 8596469e-37df-4d9e-9793-d238238131f5, log id: 3021149c >2014-07-25 13:23:55,754 INFO [org.ovirt.engine.core.bll.CommandMultiAsyncTasks] (DefaultQuartzScheduler_Worker-13) [4dca0641] CommandMultiAsyncTasks::AttachTask: Attaching task 4eccf3c4-6647-4bbe-b4ae-eb6a84f8d40f to command 8c80d80c-39bb-4109-a465-c2cb1c952b15. >2014-07-25 13:23:55,762 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (DefaultQuartzScheduler_Worker-13) [4dca0641] Adding task 4eccf3c4-6647-4bbe-b4ae-eb6a84f8d40f (Parent Command CreateOvfStoresForStorageDomain, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters), polling hasn't started yet.. >2014-07-25 13:23:55,771 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-13) [4dca0641] Correlation ID: 665a234b, Call Stack: null, Custom Event ID: -1, Message: Add-Disk operation of 'OVF_STORE' was initiated by Non interactive user. >2014-07-25 13:23:55,774 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (DefaultQuartzScheduler_Worker-13) [4dca0641] BaseAsyncTask::startPollingTask: Starting to poll task fc24cd46-4312-49e7-adda-5903f75e3e77. >2014-07-25 13:23:55,775 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (DefaultQuartzScheduler_Worker-13) [4dca0641] BaseAsyncTask::startPollingTask: Starting to poll task 4eccf3c4-6647-4bbe-b4ae-eb6a84f8d40f. >2014-07-25 13:23:55,776 INFO [org.ovirt.engine.core.bll.ProcessOvfUpdateForStorageDomainCommand] (DefaultQuartzScheduler_Worker-13) [4dca0641] Lock freed to object EngineLock [exclusiveLocks= key: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 value: STORAGE >, sharedLocks= key: 00000002-0002-0002-0002-0000000001d9 value: OVF_UPDATE >] >2014-07-25 13:23:58,692 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (DefaultQuartzScheduler_Worker-27) Polling and updating Async Tasks: 2 tasks, 2 tasks to poll now >2014-07-25 13:23:58,698 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (DefaultQuartzScheduler_Worker-27) SPMAsyncTask::PollTask: Polling task 4eccf3c4-6647-4bbe-b4ae-eb6a84f8d40f (Parent Command CreateOvfStoresForStorageDomain, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) returned status running. >2014-07-25 13:23:58,699 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (DefaultQuartzScheduler_Worker-27) SPMAsyncTask::PollTask: Polling task fc24cd46-4312-49e7-adda-5903f75e3e77 (Parent Command CreateOvfStoresForStorageDomain, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) returned status running. >2014-07-25 13:23:58,701 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (DefaultQuartzScheduler_Worker-27) Finished polling Tasks, will poll again in 10 seconds. >2014-07-25 13:24:08,708 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (DefaultQuartzScheduler_Worker-37) SPMAsyncTask::PollTask: Polling task 4eccf3c4-6647-4bbe-b4ae-eb6a84f8d40f (Parent Command CreateOvfStoresForStorageDomain, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) returned status finished, result 'success'. >2014-07-25 13:24:08,710 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (DefaultQuartzScheduler_Worker-37) BaseAsyncTask::OnTaskEndSuccess: Task 4eccf3c4-6647-4bbe-b4ae-eb6a84f8d40f (Parent Command CreateOvfStoresForStorageDomain, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) ended successfully. >2014-07-25 13:24:08,713 INFO [org.ovirt.engine.core.bll.CommandMultiAsyncTasks] (DefaultQuartzScheduler_Worker-37) Task with DB Task ID 946ebb38-af59-4f05-9376-2dc47abe3f46 and VDSM Task ID fc24cd46-4312-49e7-adda-5903f75e3e77 is in state Polling. End action for command 8c80d80c-39bb-4109-a465-c2cb1c952b15 will proceed when all the entitys tasks are completed. >2014-07-25 13:24:08,715 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (DefaultQuartzScheduler_Worker-37) SPMAsyncTask::PollTask: Polling task fc24cd46-4312-49e7-adda-5903f75e3e77 (Parent Command CreateOvfStoresForStorageDomain, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) returned status finished, result 'success'. >2014-07-25 13:24:08,718 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (DefaultQuartzScheduler_Worker-37) BaseAsyncTask::OnTaskEndSuccess: Task fc24cd46-4312-49e7-adda-5903f75e3e77 (Parent Command CreateOvfStoresForStorageDomain, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) ended successfully. >2014-07-25 13:24:08,721 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (DefaultQuartzScheduler_Worker-37) CommandAsyncTask::EndActionIfNecessary: All tasks of command 8c80d80c-39bb-4109-a465-c2cb1c952b15 has ended -> executing endAction >2014-07-25 13:24:08,722 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (DefaultQuartzScheduler_Worker-37) CommandAsyncTask::endAction: Ending action for 2 tasks (command ID: 8c80d80c-39bb-4109-a465-c2cb1c952b15): calling endAction . >2014-07-25 13:24:08,724 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-8-thread-20) CommandAsyncTask::EndCommandAction [within thread] context: Attempting to endAction CreateOvfStoresForStorageDomain, executionIndex: 0 >2014-07-25 13:24:08,726 INFO [org.ovirt.engine.core.bll.CreateOvfStoresForStorageDomainCommand] (org.ovirt.thread.pool-8-thread-20) [4168ff2b] Ending command successfully: org.ovirt.engine.core.bll.CreateOvfStoresForStorageDomainCommand >2014-07-25 13:24:08,739 INFO [org.ovirt.engine.core.bll.AddImageFromScratchCommand] (org.ovirt.thread.pool-8-thread-20) [63bc3d2f] Ending command successfully: org.ovirt.engine.core.bll.AddImageFromScratchCommand >2014-07-25 13:24:08,742 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetImageInfoVDSCommand] (org.ovirt.thread.pool-8-thread-20) [63bc3d2f] START, GetImageInfoVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, storageDomainId = 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752, imageGroupId = 07c061c2-77f4-4c02-8402-e4dff773b216, imageId = 1f5f82d1-85c2-4386-8764-02bab6dcbc21), log id: 1a694bb9 >2014-07-25 13:24:08,771 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetImageInfoVDSCommand] (org.ovirt.thread.pool-8-thread-20) [63bc3d2f] FINISH, GetImageInfoVDSCommand, return: org.ovirt.engine.core.common.businessentities.DiskImage@c25fab30, log id: 1a694bb9 >2014-07-25 13:24:08,794 INFO [org.ovirt.engine.core.bll.AddImageFromScratchCommand] (org.ovirt.thread.pool-8-thread-20) [4dca0641] Ending command successfully: org.ovirt.engine.core.bll.AddImageFromScratchCommand >2014-07-25 13:24:08,797 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetImageInfoVDSCommand] (org.ovirt.thread.pool-8-thread-20) [4dca0641] START, GetImageInfoVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, storageDomainId = 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752, imageGroupId = 90108fe1-56a4-4f0e-b301-fa0ef46ba717, imageId = 8596469e-37df-4d9e-9793-d238238131f5), log id: 6c3de6b5 >2014-07-25 13:24:08,827 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetImageInfoVDSCommand] (org.ovirt.thread.pool-8-thread-20) [4dca0641] FINISH, GetImageInfoVDSCommand, return: org.ovirt.engine.core.common.businessentities.DiskImage@67a79d02, log id: 6c3de6b5 >2014-07-25 13:24:08,871 INFO [org.ovirt.engine.core.bll.ProcessOvfUpdateForStorageDomainCommand] (org.ovirt.thread.pool-8-thread-20) [4168ff2b] Running command: ProcessOvfUpdateForStorageDomainCommand internal: true. Entities affected : ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage >2014-07-25 13:24:08,941 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (org.ovirt.thread.pool-8-thread-20) [4168ff2b] START, SetVolumeDescriptionVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, storageDomainId = 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752, imageGroupId = 90108fe1-56a4-4f0e-b301-fa0ef46ba717, imageId = 8596469e-37df-4d9e-9793-d238238131f5), log id: 4e546ca7 >2014-07-25 13:24:08,942 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (org.ovirt.thread.pool-8-thread-20) [4168ff2b] -- executeIrsBrokerCommand: calling 'setVolumeDescription' >2014-07-25 13:24:08,946 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (org.ovirt.thread.pool-8-thread-20) [4168ff2b] -- setVolumeDescription parameters: > spUUID=00000002-0002-0002-0002-0000000001d9 > sdUUID=71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 > imageGroupGUID=90108fe1-56a4-4f0e-b301-fa0ef46ba717 > volUUID=8596469e-37df-4d9e-9793-d238238131f5 > description={"Updated":false,"Disk Description":"OVF_STORE","Storage Domains":[{"uuid":"71f82fcd-4fa6-4f56-b228-f4fa4bd4f752"}],"Last Updated":null} > >2014-07-25 13:24:09,105 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (org.ovirt.thread.pool-8-thread-20) [4168ff2b] FINISH, SetVolumeDescriptionVDSCommand, log id: 4e546ca7 >2014-07-25 13:24:09,127 INFO [org.ovirt.engine.core.bll.UploadStreamCommand] (org.ovirt.thread.pool-8-thread-20) [30258eda] Lock Acquired to object EngineLock [exclusiveLocks= , sharedLocks= key: 17f7fe56-5f32-40ef-b098-611b84484e0b value: VDS_EXECUTION >] >2014-07-25 13:24:09,131 INFO [org.ovirt.engine.core.bll.UploadStreamCommand] (org.ovirt.thread.pool-8-thread-20) [30258eda] Running command: UploadStreamCommand internal: true. Entities affected : ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage >2014-07-25 13:24:09,139 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.UploadStreamVDSCommand] (org.ovirt.thread.pool-8-thread-20) [30258eda] START, UploadStreamVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 3afb9ecf >2014-07-25 13:24:09,140 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.UploadStreamVDSCommand] (org.ovirt.thread.pool-8-thread-20) [30258eda] -- executeVdsBrokerCommand: >2014-07-25 13:24:09,142 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.UploadStreamVDSCommand] (org.ovirt.thread.pool-8-thread-20) [30258eda] -- parameters: > spUUID=00000002-0002-0002-0002-0000000001d9 > sdUUID=71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 > imageGUID=90108fe1-56a4-4f0e-b301-fa0ef46ba717 > volUUID=8596469e-37df-4d9e-9793-d238238131f5 > size=10,240 > >2014-07-25 13:24:09,996 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.UploadStreamVDSCommand] (org.ovirt.thread.pool-8-thread-20) [30258eda] FINISH, UploadStreamVDSCommand, log id: 3afb9ecf >2014-07-25 13:24:10,012 INFO [org.ovirt.engine.core.bll.CommandMultiAsyncTasks] (org.ovirt.thread.pool-8-thread-20) [30258eda] CommandMultiAsyncTasks::AttachTask: Attaching task b1a6bc7b-1427-405f-8d32-41e4358fd139 to command 8c80d80c-39bb-4109-a465-c2cb1c952b15. >2014-07-25 13:24:10,022 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (org.ovirt.thread.pool-8-thread-20) [30258eda] Adding task b1a6bc7b-1427-405f-8d32-41e4358fd139 (Parent Command ProcessOvfUpdateForStorageDomain, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters), polling hasn't started yet.. >2014-07-25 13:24:10,024 INFO [org.ovirt.engine.core.bll.UploadStreamCommand] (org.ovirt.thread.pool-8-thread-20) [30258eda] Lock freed to object EngineLock [exclusiveLocks= , sharedLocks= key: 17f7fe56-5f32-40ef-b098-611b84484e0b value: VDS_EXECUTION >] >2014-07-25 13:24:10,034 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (org.ovirt.thread.pool-8-thread-20) [30258eda] START, SetVolumeDescriptionVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, storageDomainId = 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752, imageGroupId = 90108fe1-56a4-4f0e-b301-fa0ef46ba717, imageId = 8596469e-37df-4d9e-9793-d238238131f5), log id: 577c34e1 >2014-07-25 13:24:10,037 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (org.ovirt.thread.pool-8-thread-20) [30258eda] -- executeIrsBrokerCommand: calling 'setVolumeDescription' >2014-07-25 13:24:10,038 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (org.ovirt.thread.pool-8-thread-20) [30258eda] -- setVolumeDescription parameters: > spUUID=00000002-0002-0002-0002-0000000001d9 > sdUUID=71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 > imageGroupGUID=90108fe1-56a4-4f0e-b301-fa0ef46ba717 > volUUID=8596469e-37df-4d9e-9793-d238238131f5 > description={"Updated":true,"Disk Description":"OVF_STORE","Storage Domains":[{"uuid":"71f82fcd-4fa6-4f56-b228-f4fa4bd4f752"}],"Last Updated":"Fri Jul 25 13:24:08 CEST 2014","Size":10240} > >2014-07-25 13:24:10,340 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (org.ovirt.thread.pool-8-thread-20) [30258eda] FINISH, SetVolumeDescriptionVDSCommand, log id: 577c34e1 >2014-07-25 13:24:10,349 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (org.ovirt.thread.pool-8-thread-20) [30258eda] START, SetVolumeDescriptionVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, storageDomainId = 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752, imageGroupId = 07c061c2-77f4-4c02-8402-e4dff773b216, imageId = 1f5f82d1-85c2-4386-8764-02bab6dcbc21), log id: 488ca852 >2014-07-25 13:24:10,352 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (org.ovirt.thread.pool-8-thread-20) [30258eda] -- executeIrsBrokerCommand: calling 'setVolumeDescription' >2014-07-25 13:24:10,353 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (org.ovirt.thread.pool-8-thread-20) [30258eda] -- setVolumeDescription parameters: > spUUID=00000002-0002-0002-0002-0000000001d9 > sdUUID=71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 > imageGroupGUID=07c061c2-77f4-4c02-8402-e4dff773b216 > volUUID=1f5f82d1-85c2-4386-8764-02bab6dcbc21 > description={"Updated":false,"Disk Description":"OVF_STORE","Storage Domains":[{"uuid":"71f82fcd-4fa6-4f56-b228-f4fa4bd4f752"}],"Last Updated":null} > >2014-07-25 13:24:10,523 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (org.ovirt.thread.pool-8-thread-20) [30258eda] FINISH, SetVolumeDescriptionVDSCommand, log id: 488ca852 >2014-07-25 13:24:10,539 INFO [org.ovirt.engine.core.bll.UploadStreamCommand] (org.ovirt.thread.pool-8-thread-20) [3c1435c2] Lock Acquired to object EngineLock [exclusiveLocks= , sharedLocks= key: 17f7fe56-5f32-40ef-b098-611b84484e0b value: VDS_EXECUTION >] >2014-07-25 13:24:10,542 INFO [org.ovirt.engine.core.bll.UploadStreamCommand] (org.ovirt.thread.pool-8-thread-20) [3c1435c2] Running command: UploadStreamCommand internal: true. Entities affected : ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage >2014-07-25 13:24:10,544 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.UploadStreamVDSCommand] (org.ovirt.thread.pool-8-thread-20) [3c1435c2] START, UploadStreamVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 73547d41 >2014-07-25 13:24:10,545 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.UploadStreamVDSCommand] (org.ovirt.thread.pool-8-thread-20) [3c1435c2] -- executeVdsBrokerCommand: >2014-07-25 13:24:10,546 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.UploadStreamVDSCommand] (org.ovirt.thread.pool-8-thread-20) [3c1435c2] -- parameters: > spUUID=00000002-0002-0002-0002-0000000001d9 > sdUUID=71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 > imageGUID=07c061c2-77f4-4c02-8402-e4dff773b216 > volUUID=1f5f82d1-85c2-4386-8764-02bab6dcbc21 > size=10,240 > >2014-07-25 13:24:11,234 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.UploadStreamVDSCommand] (org.ovirt.thread.pool-8-thread-20) [3c1435c2] FINISH, UploadStreamVDSCommand, log id: 73547d41 >2014-07-25 13:24:11,236 INFO [org.ovirt.engine.core.bll.CommandMultiAsyncTasks] (org.ovirt.thread.pool-8-thread-20) [3c1435c2] CommandMultiAsyncTasks::AttachTask: Attaching task bbc204bf-5244-41ff-bf18-1de1ce9966ba to command 8c80d80c-39bb-4109-a465-c2cb1c952b15. >2014-07-25 13:24:11,243 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (org.ovirt.thread.pool-8-thread-20) [3c1435c2] Adding task bbc204bf-5244-41ff-bf18-1de1ce9966ba (Parent Command ProcessOvfUpdateForStorageDomain, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters), polling hasn't started yet.. >2014-07-25 13:24:11,245 INFO [org.ovirt.engine.core.bll.UploadStreamCommand] (org.ovirt.thread.pool-8-thread-20) [3c1435c2] Lock freed to object EngineLock [exclusiveLocks= , sharedLocks= key: 17f7fe56-5f32-40ef-b098-611b84484e0b value: VDS_EXECUTION >] >2014-07-25 13:24:11,249 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (org.ovirt.thread.pool-8-thread-20) [3c1435c2] START, SetVolumeDescriptionVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, storageDomainId = 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752, imageGroupId = 07c061c2-77f4-4c02-8402-e4dff773b216, imageId = 1f5f82d1-85c2-4386-8764-02bab6dcbc21), log id: 40b7a99 >2014-07-25 13:24:11,252 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (org.ovirt.thread.pool-8-thread-20) [3c1435c2] -- executeIrsBrokerCommand: calling 'setVolumeDescription' >2014-07-25 13:24:11,253 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (org.ovirt.thread.pool-8-thread-20) [3c1435c2] -- setVolumeDescription parameters: > spUUID=00000002-0002-0002-0002-0000000001d9 > sdUUID=71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 > imageGroupGUID=07c061c2-77f4-4c02-8402-e4dff773b216 > volUUID=1f5f82d1-85c2-4386-8764-02bab6dcbc21 > description={"Updated":true,"Disk Description":"OVF_STORE","Storage Domains":[{"uuid":"71f82fcd-4fa6-4f56-b228-f4fa4bd4f752"}],"Last Updated":"Fri Jul 25 13:24:08 CEST 2014","Size":10240} > >2014-07-25 13:24:11,453 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (org.ovirt.thread.pool-8-thread-20) [3c1435c2] FINISH, SetVolumeDescriptionVDSCommand, log id: 40b7a99 >2014-07-25 13:24:11,457 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (org.ovirt.thread.pool-8-thread-20) [3c1435c2] BaseAsyncTask::startPollingTask: Starting to poll task b1a6bc7b-1427-405f-8d32-41e4358fd139. >2014-07-25 13:24:11,458 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (org.ovirt.thread.pool-8-thread-20) [3c1435c2] BaseAsyncTask::startPollingTask: Starting to poll task bbc204bf-5244-41ff-bf18-1de1ce9966ba. >2014-07-25 13:24:11,459 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-8-thread-20) [3c1435c2] CommandAsyncTask::HandleEndActionResult [within thread]: endAction for action type CreateOvfStoresForStorageDomain completed, handling the result. >2014-07-25 13:24:11,460 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-8-thread-20) [3c1435c2] CommandAsyncTask::HandleEndActionResult [within thread]: endAction for action type CreateOvfStoresForStorageDomain succeeded, clearing tasks. >2014-07-25 13:24:11,462 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (org.ovirt.thread.pool-8-thread-20) [3c1435c2] SPMAsyncTask::ClearAsyncTask: Attempting to clear task fc24cd46-4312-49e7-adda-5903f75e3e77 >2014-07-25 13:24:11,463 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-20) [3c1435c2] START, SPMClearTaskVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, taskId = fc24cd46-4312-49e7-adda-5903f75e3e77), log id: 73f70cc7 >2014-07-25 13:24:11,465 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-20) [3c1435c2] START, HSMClearTaskVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, taskId=fc24cd46-4312-49e7-adda-5903f75e3e77), log id: 6068184b >2014-07-25 13:24:11,606 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-20) [3c1435c2] FINISH, HSMClearTaskVDSCommand, log id: 6068184b >2014-07-25 13:24:11,607 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-20) [3c1435c2] FINISH, SPMClearTaskVDSCommand, log id: 73f70cc7 >2014-07-25 13:24:11,613 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (org.ovirt.thread.pool-8-thread-20) [3c1435c2] BaseAsyncTask::RemoveTaskFromDB: Removed task fc24cd46-4312-49e7-adda-5903f75e3e77 from DataBase >2014-07-25 13:24:11,614 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (org.ovirt.thread.pool-8-thread-20) [3c1435c2] SPMAsyncTask::ClearAsyncTask: Attempting to clear task 4eccf3c4-6647-4bbe-b4ae-eb6a84f8d40f >2014-07-25 13:24:11,616 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-20) [3c1435c2] START, SPMClearTaskVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, taskId = 4eccf3c4-6647-4bbe-b4ae-eb6a84f8d40f), log id: 3bfddcbc >2014-07-25 13:24:11,618 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-20) [3c1435c2] START, HSMClearTaskVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, taskId=4eccf3c4-6647-4bbe-b4ae-eb6a84f8d40f), log id: 17a6d969 >2014-07-25 13:24:11,708 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-20) [3c1435c2] FINISH, HSMClearTaskVDSCommand, log id: 17a6d969 >2014-07-25 13:24:11,709 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-20) [3c1435c2] FINISH, SPMClearTaskVDSCommand, log id: 3bfddcbc >2014-07-25 13:24:11,714 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (org.ovirt.thread.pool-8-thread-20) [3c1435c2] BaseAsyncTask::RemoveTaskFromDB: Removed task 4eccf3c4-6647-4bbe-b4ae-eb6a84f8d40f from DataBase >2014-07-25 13:24:11,716 INFO [org.ovirt.engine.core.bll.CommandMultiAsyncTasks] (org.ovirt.thread.pool-8-thread-20) [3c1435c2] [within thread]: Some of the tasks related to command id 8c80d80c-39bb-4109-a465-c2cb1c952b15 were not cleared yet (Task id bbc204bf-5244-41ff-bf18-1de1ce9966ba is in state Polling). >2014-07-25 13:24:18,725 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (DefaultQuartzScheduler_Worker-64) Polling and updating Async Tasks: 4 tasks, 2 tasks to poll now >2014-07-25 13:24:18,731 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (DefaultQuartzScheduler_Worker-64) SPMAsyncTask::PollTask: Polling task b1a6bc7b-1427-405f-8d32-41e4358fd139 (Parent Command ProcessOvfUpdateForStorageDomain, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) returned status finished, result 'success'. >2014-07-25 13:24:18,733 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (DefaultQuartzScheduler_Worker-64) BaseAsyncTask::OnTaskEndSuccess: Task b1a6bc7b-1427-405f-8d32-41e4358fd139 (Parent Command ProcessOvfUpdateForStorageDomain, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) ended successfully. >2014-07-25 13:24:18,734 INFO [org.ovirt.engine.core.bll.CommandMultiAsyncTasks] (DefaultQuartzScheduler_Worker-64) Task with DB Task ID 3aad0be1-afd6-4131-8135-7425d953b511 and VDSM Task ID bbc204bf-5244-41ff-bf18-1de1ce9966ba is in state Polling. End action for command 8c80d80c-39bb-4109-a465-c2cb1c952b15 will proceed when all the entitys tasks are completed. >2014-07-25 13:24:18,736 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (DefaultQuartzScheduler_Worker-64) SPMAsyncTask::PollTask: Polling task bbc204bf-5244-41ff-bf18-1de1ce9966ba (Parent Command ProcessOvfUpdateForStorageDomain, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) returned status finished, result 'success'. >2014-07-25 13:24:18,737 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (DefaultQuartzScheduler_Worker-64) BaseAsyncTask::OnTaskEndSuccess: Task bbc204bf-5244-41ff-bf18-1de1ce9966ba (Parent Command ProcessOvfUpdateForStorageDomain, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) ended successfully. >2014-07-25 13:24:18,739 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (DefaultQuartzScheduler_Worker-64) CommandAsyncTask::EndActionIfNecessary: All tasks of command 8c80d80c-39bb-4109-a465-c2cb1c952b15 has ended -> executing endAction >2014-07-25 13:24:18,740 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (DefaultQuartzScheduler_Worker-64) CommandAsyncTask::endAction: Ending action for 2 tasks (command ID: 8c80d80c-39bb-4109-a465-c2cb1c952b15): calling endAction . >2014-07-25 13:24:18,741 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-8-thread-47) CommandAsyncTask::EndCommandAction [within thread] context: Attempting to endAction ProcessOvfUpdateForStorageDomain, executionIndex: 0 >2014-07-25 13:24:18,771 INFO [org.ovirt.engine.core.bll.ProcessOvfUpdateForStorageDomainCommand] (org.ovirt.thread.pool-8-thread-47) [4168ff2b] Ending command successfully: org.ovirt.engine.core.bll.ProcessOvfUpdateForStorageDomainCommand >2014-07-25 13:24:18,773 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-8-thread-47) [4168ff2b] CommandAsyncTask::HandleEndActionResult [within thread]: endAction for action type ProcessOvfUpdateForStorageDomain completed, handling the result. >2014-07-25 13:24:18,774 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-8-thread-47) [4168ff2b] CommandAsyncTask::HandleEndActionResult [within thread]: endAction for action type ProcessOvfUpdateForStorageDomain succeeded, clearing tasks. >2014-07-25 13:24:18,775 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (org.ovirt.thread.pool-8-thread-47) [4168ff2b] SPMAsyncTask::ClearAsyncTask: Attempting to clear task bbc204bf-5244-41ff-bf18-1de1ce9966ba >2014-07-25 13:24:18,776 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-47) [4168ff2b] START, SPMClearTaskVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, taskId = bbc204bf-5244-41ff-bf18-1de1ce9966ba), log id: 602cc445 >2014-07-25 13:24:18,779 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-47) [4168ff2b] START, HSMClearTaskVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, taskId=bbc204bf-5244-41ff-bf18-1de1ce9966ba), log id: 1b09bf57 >2014-07-25 13:24:18,813 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-47) [4168ff2b] FINISH, HSMClearTaskVDSCommand, log id: 1b09bf57 >2014-07-25 13:24:18,814 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-47) [4168ff2b] FINISH, SPMClearTaskVDSCommand, log id: 602cc445 >2014-07-25 13:24:18,832 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (org.ovirt.thread.pool-8-thread-47) [4168ff2b] BaseAsyncTask::RemoveTaskFromDB: Removed task bbc204bf-5244-41ff-bf18-1de1ce9966ba from DataBase >2014-07-25 13:24:18,833 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (org.ovirt.thread.pool-8-thread-47) [4168ff2b] SPMAsyncTask::ClearAsyncTask: Attempting to clear task b1a6bc7b-1427-405f-8d32-41e4358fd139 >2014-07-25 13:24:18,835 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-47) [4168ff2b] START, SPMClearTaskVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, taskId = b1a6bc7b-1427-405f-8d32-41e4358fd139), log id: 7b857e7b >2014-07-25 13:24:18,837 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-47) [4168ff2b] START, HSMClearTaskVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, taskId=b1a6bc7b-1427-405f-8d32-41e4358fd139), log id: 7932fc97 >2014-07-25 13:24:18,918 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-47) [4168ff2b] FINISH, HSMClearTaskVDSCommand, log id: 7932fc97 >2014-07-25 13:24:18,919 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-47) [4168ff2b] FINISH, SPMClearTaskVDSCommand, log id: 7b857e7b >2014-07-25 13:24:18,924 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (org.ovirt.thread.pool-8-thread-47) [4168ff2b] BaseAsyncTask::RemoveTaskFromDB: Removed task b1a6bc7b-1427-405f-8d32-41e4358fd139 from DataBase >2014-07-25 13:24:18,926 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-8-thread-47) [4168ff2b] CommandAsyncTask::HandleEndActionResult [within thread]: Removing CommandMultiAsyncTasks object for entity 8c80d80c-39bb-4109-a465-c2cb1c952b15 >2014-07-25 13:25:24,278 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (DefaultQuartzScheduler_Worker-18) Setting new tasks map. The map contains now 0 tasks >2014-07-25 13:25:24,279 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (DefaultQuartzScheduler_Worker-18) Cleared all tasks of pool 00000002-0002-0002-0002-0000000001d9. >2014-07-25 13:34:11,880 INFO [org.ovirt.engine.core.bll.AddVmFromScratchCommand] (ajp--127.0.0.1-8702-9) [76d45966] Lock Acquired to object EngineLock [exclusiveLocks= key: testtesttest value: VM_NAME >, sharedLocks= ] >2014-07-25 13:34:11,932 INFO [org.ovirt.engine.core.bll.AddVmFromScratchCommand] (ajp--127.0.0.1-8702-9) [76d45966] Running command: AddVmFromScratchCommand internal: false. Entities affected : ID: 00000001-0001-0001-0001-0000000002c2 Type: VdsGroups >2014-07-25 13:34:12,053 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-9) [76d45966] Correlation ID: 76d45966, Job ID: 24d01812-2438-4a5f-b1f9-5e96401f78f9, Call Stack: null, Custom Event ID: -1, Message: VM testtesttest was created by admin. >2014-07-25 13:34:12,068 INFO [org.ovirt.engine.core.bll.AddVmFromScratchCommand] (ajp--127.0.0.1-8702-9) [76d45966] Lock freed to object EngineLock [exclusiveLocks= key: testtesttest value: VM_NAME >, sharedLocks= ] >2014-07-25 13:34:21,162 INFO [org.ovirt.engine.core.bll.network.vm.AddVmInterfaceCommand] (ajp--127.0.0.1-8702-10) [d9e7c59] Running command: AddVmInterfaceCommand internal: false. Entities affected : ID: 78722009-2727-4715-91bf-dbbb33ce111e Type: VM, ID: 0000000e-000e-000e-000e-0000000003f5 Type: VnicProfile >2014-07-25 13:34:21,211 INFO [org.ovirt.engine.core.bll.network.vm.ActivateDeactivateVmNicCommand] (ajp--127.0.0.1-8702-10) [3a8e225] Running command: ActivateDeactivateVmNicCommand internal: true. Entities affected : ID: 78722009-2727-4715-91bf-dbbb33ce111e Type: VM >2014-07-25 13:34:21,262 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-10) [3a8e225] Correlation ID: 3a8e225, Call Stack: null, Custom Event ID: -1, Message: Network Interface nic1 (VirtIO) was plugged to VM testtesttest. (User: admin) >2014-07-25 13:34:21,273 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-10) [3a8e225] Correlation ID: d9e7c59, Call Stack: null, Custom Event ID: -1, Message: Interface nic1 (VirtIO) was added to VM testtesttest. (User: admin) >2014-07-25 13:34:21,697 INFO [org.ovirt.engine.core.bll.network.vm.ReorderVmNicsCommand] (ajp--127.0.0.1-8702-7) [5e069bea] Running command: ReorderVmNicsCommand internal: false. Entities affected : ID: 78722009-2727-4715-91bf-dbbb33ce111e Type: VM >2014-07-25 13:34:26,154 INFO [org.ovirt.engine.core.bll.AddDiskCommand] (ajp--127.0.0.1-8702-2) [728fbda2] Lock Acquired to object EngineLock [exclusiveLocks= key: 78722009-2727-4715-91bf-dbbb33ce111e value: VM_DISK_BOOT >, sharedLocks= key: 78722009-2727-4715-91bf-dbbb33ce111e value: VM >] >2014-07-25 13:34:26,217 INFO [org.ovirt.engine.core.bll.AddDiskCommand] (ajp--127.0.0.1-8702-2) [728fbda2] Running command: AddDiskCommand internal: false. Entities affected : ID: 78722009-2727-4715-91bf-dbbb33ce111e Type: VM, ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage >2014-07-25 13:34:26,259 INFO [org.ovirt.engine.core.bll.AddImageFromScratchCommand] (ajp--127.0.0.1-8702-2) [43def421] Running command: AddImageFromScratchCommand internal: true. Entities affected : ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage >2014-07-25 13:34:26,274 INFO [org.ovirt.engine.core.bll.AddImageFromScratchCommand] (ajp--127.0.0.1-8702-2) [43def421] Lock freed to object EngineLock [exclusiveLocks= key: 78722009-2727-4715-91bf-dbbb33ce111e value: VM_DISK_BOOT >, sharedLocks= key: 78722009-2727-4715-91bf-dbbb33ce111e value: VM >] >2014-07-25 13:34:26,276 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.CreateImageVDSCommand] (ajp--127.0.0.1-8702-2) [43def421] START, CreateImageVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, storageDomainId = 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752, imageGroupId = 6cae69b6-2d4f-4e61-993c-dce113dd19e5, imageSizeInBytes = 21474836480, volumeFormat = RAW, newImageId = b05d9fa4-2cf0-414e-b031-f89fcdf2cf0b, newImageDescription = ), log id: 7d691e51 >2014-07-25 13:34:26,282 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.CreateImageVDSCommand] (ajp--127.0.0.1-8702-2) [43def421] -- executeIrsBrokerCommand: calling 'createVolume' with two new parameters: description and UUID >2014-07-25 13:34:26,662 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.CreateImageVDSCommand] (ajp--127.0.0.1-8702-2) [43def421] FINISH, CreateImageVDSCommand, return: b05d9fa4-2cf0-414e-b031-f89fcdf2cf0b, log id: 7d691e51 >2014-07-25 13:34:26,671 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (ajp--127.0.0.1-8702-2) [43def421] CommandAsyncTask::Adding CommandMultiAsyncTasks object for command 0500fc29-af6c-44ff-be85-2f60df4cf00a >2014-07-25 13:34:26,672 INFO [org.ovirt.engine.core.bll.CommandMultiAsyncTasks] (ajp--127.0.0.1-8702-2) [43def421] CommandMultiAsyncTasks::AttachTask: Attaching task cbda6151-cac6-46ea-af0c-206457c6cc2b to command 0500fc29-af6c-44ff-be85-2f60df4cf00a. >2014-07-25 13:34:26,688 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (ajp--127.0.0.1-8702-2) [43def421] Adding task cbda6151-cac6-46ea-af0c-206457c6cc2b (Parent Command AddDisk, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters), polling hasn't started yet.. >2014-07-25 13:34:26,717 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-2) [43def421] Correlation ID: 728fbda2, Job ID: fba17df4-f9d0-4038-aa71-316111e063ca, Call Stack: null, Custom Event ID: -1, Message: Add-Disk operation of testtesttest_Disk1 was initiated on VM testtesttest by admin. >2014-07-25 13:34:26,719 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (ajp--127.0.0.1-8702-2) [43def421] BaseAsyncTask::startPollingTask: Starting to poll task cbda6151-cac6-46ea-af0c-206457c6cc2b. >2014-07-25 13:34:28,763 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (DefaultQuartzScheduler_Worker-52) Polling and updating Async Tasks: 1 tasks, 1 tasks to poll now >2014-07-25 13:34:28,770 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (DefaultQuartzScheduler_Worker-52) SPMAsyncTask::PollTask: Polling task cbda6151-cac6-46ea-af0c-206457c6cc2b (Parent Command AddDisk, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) returned status finished, result 'success'. >2014-07-25 13:34:28,776 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (DefaultQuartzScheduler_Worker-52) BaseAsyncTask::OnTaskEndSuccess: Task cbda6151-cac6-46ea-af0c-206457c6cc2b (Parent Command AddDisk, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) ended successfully. >2014-07-25 13:34:28,778 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (DefaultQuartzScheduler_Worker-52) CommandAsyncTask::EndActionIfNecessary: All tasks of command 0500fc29-af6c-44ff-be85-2f60df4cf00a has ended -> executing endAction >2014-07-25 13:34:28,779 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (DefaultQuartzScheduler_Worker-52) CommandAsyncTask::endAction: Ending action for 1 tasks (command ID: 0500fc29-af6c-44ff-be85-2f60df4cf00a): calling endAction . >2014-07-25 13:34:28,780 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-8-thread-49) CommandAsyncTask::EndCommandAction [within thread] context: Attempting to endAction AddDisk, executionIndex: 0 >2014-07-25 13:34:28,791 INFO [org.ovirt.engine.core.bll.AddDiskCommand] (org.ovirt.thread.pool-8-thread-49) [728fbda2] Ending command successfully: org.ovirt.engine.core.bll.AddDiskCommand >2014-07-25 13:34:28,798 INFO [org.ovirt.engine.core.bll.AddImageFromScratchCommand] (org.ovirt.thread.pool-8-thread-49) [43def421] Ending command successfully: org.ovirt.engine.core.bll.AddImageFromScratchCommand >2014-07-25 13:34:28,808 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetImageInfoVDSCommand] (org.ovirt.thread.pool-8-thread-49) [43def421] START, GetImageInfoVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, storageDomainId = 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752, imageGroupId = 6cae69b6-2d4f-4e61-993c-dce113dd19e5, imageId = b05d9fa4-2cf0-414e-b031-f89fcdf2cf0b), log id: 3f76efbe >2014-07-25 13:34:28,833 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetImageInfoVDSCommand] (org.ovirt.thread.pool-8-thread-49) [43def421] FINISH, GetImageInfoVDSCommand, return: org.ovirt.engine.core.common.businessentities.DiskImage@fd653795, log id: 3f76efbe >2014-07-25 13:34:28,855 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-49) Correlation ID: 728fbda2, Call Stack: null, Custom Event ID: -1, Message: The disk testtesttest_Disk1 was successfully added to VM testtesttest. >2014-07-25 13:34:28,856 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-8-thread-49) CommandAsyncTask::HandleEndActionResult [within thread]: endAction for action type AddDisk completed, handling the result. >2014-07-25 13:34:28,857 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-8-thread-49) CommandAsyncTask::HandleEndActionResult [within thread]: endAction for action type AddDisk succeeded, clearing tasks. >2014-07-25 13:34:28,862 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (org.ovirt.thread.pool-8-thread-49) SPMAsyncTask::ClearAsyncTask: Attempting to clear task cbda6151-cac6-46ea-af0c-206457c6cc2b >2014-07-25 13:34:28,863 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-49) START, SPMClearTaskVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, taskId = cbda6151-cac6-46ea-af0c-206457c6cc2b), log id: 6fdb8433 >2014-07-25 13:34:28,866 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-49) START, HSMClearTaskVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, taskId=cbda6151-cac6-46ea-af0c-206457c6cc2b), log id: 34c469a4 >2014-07-25 13:34:28,915 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-49) FINISH, HSMClearTaskVDSCommand, log id: 34c469a4 >2014-07-25 13:34:28,917 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-49) FINISH, SPMClearTaskVDSCommand, log id: 6fdb8433 >2014-07-25 13:34:28,928 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (org.ovirt.thread.pool-8-thread-49) BaseAsyncTask::RemoveTaskFromDB: Removed task cbda6151-cac6-46ea-af0c-206457c6cc2b from DataBase >2014-07-25 13:34:28,930 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-8-thread-49) CommandAsyncTask::HandleEndActionResult [within thread]: Removing CommandMultiAsyncTasks object for entity 0500fc29-af6c-44ff-be85-2f60df4cf00a >2014-07-25 13:34:34,885 INFO [org.ovirt.engine.core.bll.RunVmCommand] (ajp--127.0.0.1-8702-7) [605e9f69] Lock Acquired to object EngineLock [exclusiveLocks= key: a2660663-ef34-41ee-a580-0fc425183c36 value: VM >, sharedLocks= ] >2014-07-25 13:34:34,890 WARN [org.ovirt.engine.core.bll.RunVmCommand] (ajp--127.0.0.1-8702-7) [605e9f69] CanDoAction of action RunVm failed. Reasons:VAR__ACTION__RUN,VAR__TYPE__VM,ACTION_TYPE_FAILED_CANNOT_RUN_ACTION_ON_NON_MANAGED_VM >2014-07-25 13:34:34,891 INFO [org.ovirt.engine.core.bll.RunVmCommand] (ajp--127.0.0.1-8702-7) [605e9f69] Lock freed to object EngineLock [exclusiveLocks= key: a2660663-ef34-41ee-a580-0fc425183c36 value: VM >, sharedLocks= ] >2014-07-25 13:34:38,784 INFO [org.ovirt.engine.core.bll.RunVmCommand] (ajp--127.0.0.1-8702-8) [4f3adf5f] Lock Acquired to object EngineLock [exclusiveLocks= key: 78722009-2727-4715-91bf-dbbb33ce111e value: VM >, sharedLocks= ] >2014-07-25 13:34:38,821 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (ajp--127.0.0.1-8702-8) [4f3adf5f] START, IsVmDuringInitiatingVDSCommand( vmId = 78722009-2727-4715-91bf-dbbb33ce111e), log id: 39727cec >2014-07-25 13:34:38,822 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (ajp--127.0.0.1-8702-8) [4f3adf5f] FINISH, IsVmDuringInitiatingVDSCommand, return: false, log id: 39727cec >2014-07-25 13:34:38,888 INFO [org.ovirt.engine.core.bll.RunVmCommand] (org.ovirt.thread.pool-8-thread-41) [4f3adf5f] Running command: RunVmCommand internal: false. Entities affected : ID: 78722009-2727-4715-91bf-dbbb33ce111e Type: VM >2014-07-25 13:34:38,940 INFO [org.ovirt.engine.core.vdsbroker.UpdateVmDynamicDataVDSCommand] (org.ovirt.thread.pool-8-thread-41) [4f3adf5f] START, UpdateVmDynamicDataVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vmDynamic=org.ovirt.engine.core.common.businessentities.VmDynamic@f5735a74), log id: 65b1b34f >2014-07-25 13:34:38,946 INFO [org.ovirt.engine.core.vdsbroker.UpdateVmDynamicDataVDSCommand] (org.ovirt.thread.pool-8-thread-41) [4f3adf5f] FINISH, UpdateVmDynamicDataVDSCommand, log id: 65b1b34f >2014-07-25 13:34:38,960 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (org.ovirt.thread.pool-8-thread-41) [4f3adf5f] START, CreateVmVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vmId=78722009-2727-4715-91bf-dbbb33ce111e, vm=VM [testtesttest]), log id: 32bc339c >2014-07-25 13:34:38,975 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVmFromCloudInitVDSCommand] (org.ovirt.thread.pool-8-thread-41) [4f3adf5f] START, CreateVmFromCloudInitVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vmId=78722009-2727-4715-91bf-dbbb33ce111e, vm=VM [testtesttest]), log id: 4df366f >2014-07-25 13:34:39,018 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (org.ovirt.thread.pool-8-thread-41) [4f3adf5f] org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVmFromCloudInitVDSCommand kvmEnable=true,keyboardLayout=en-us,nice=0,displayNetwork=ovirtmgmt,copyPasteEnable=true,timeOffset=0,transparentHugePages=true,vmId=78722009-2727-4715-91bf-dbbb33ce111e,acpiEnable=true,custom={},spiceSslCipherSuite=DEFAULT,memSize=5120,smp=1,emulatedMachine=rhel6.5.0,vmType=kvm,memGuaranteedSize=5120,display=qxl,smartcardEnable=false,bootMenuEnable=false,spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,susbredir,ssmartcard,smpCoresPerSocket=1,numaTune={mode=preferred},maxVCpus=160,devices=[{shared=false, iface=ide, index=3, specParams={vmPayload={file={openstack/latest/meta_data.json=ewogICJsYXVuY2hfaW5kZXgiIDogIjAiLAogICJhdmFpbGFiaWxpdHlfem9uZSIgOiAibm92YSIs >CiAgIm5ldHdvcmstaW50ZXJmYWNlcyIgOiAiYXV0b1xuIiwKICAibmV0d29ya19jb25maWciIDog >ewogICAgImNvbnRlbnRfcGF0aCIgOiAiL2NvbnRlbnQvMDAwMCIsCiAgICAicGF0aCIgOiAiL2V0 >Yy9uZXR3b3JrL2ludGVyZmFjZXMiCiAgfSwKICAidXVpZCIgOiAiODM3M2I1ZmUtZjllYS00Y2Q4 >LTk0MjQtMDdmNmE5MjQzMjdjIiwKICAibWV0YSIgOiB7CiAgICAiZXNzZW50aWFsIiA6ICJmYWxz >ZSIsCiAgICAicm9sZSIgOiAic2VydmVyIiwKICAgICJkc21vZGUiIDogImxvY2FsIgogIH0KfQ== >, openstack/latest/user_data=I2Nsb3VkLWNvbmZpZwpzc2hfcHdhdXRoOiB0cnVlCmRpc2FibGVfcm9vdDogMApvdXRwdXQ6CiAg >YWxsOiAnPj4gL3Zhci9sb2cvY2xvdWQtaW5pdC1vdXRwdXQubG9nJwpjaHBhc3N3ZDoKICBleHBp >cmU6IGZhbHNlCnJ1bmNtZDoKLSAnc2VkIC1pICcnL15kYXRhc291cmNlX2xpc3Q6IC9kJycgL2V0 >Yy9jbG91ZC9jbG91ZC5jZmc7IGVjaG8gJydkYXRhc291cmNlX2xpc3Q6CiAgWyJOb0Nsb3VkIiwg >IkNvbmZpZ0RyaXZlIl0nJyA+PiAvZXRjL2Nsb3VkL2Nsb3VkLmNmZycK >, openstack/content/0000=YXV0bwo= >}, volId=config-2}}, path=, device=cdrom, type=disk, readonly=true, deviceId=407cc02a-1dee-4993-a2a1-5ef7a2e1ee34}, {specParams={vram=32768, heads=1}, device=qxl, type=video, deviceId=d962101d-abb7-4fbb-b249-2abdf8b3ee46}, {shared=false, iface=ide, index=2, specParams={path=}, path=, device=cdrom, type=disk, readonly=true, deviceId=a0db1b5f-d569-4b3f-a0e3-3c2992216d80}, {shared=false, volumeID=b05d9fa4-2cf0-414e-b031-f89fcdf2cf0b, index=0, iface=virtio, imageID=6cae69b6-2d4f-4e61-993c-dce113dd19e5, domainID=71f82fcd-4fa6-4f56-b228-f4fa4bd4f752, specParams={}, optional=false, propagateErrors=off, device=disk, poolID=00000002-0002-0002-0002-0000000001d9, format=raw, type=disk, readonly=false, deviceId=6cae69b6-2d4f-4e61-993c-dce113dd19e5}, {nicModel=pv, specParams={outbound={}, inbound={}}, macAddr=00:1a:4a:ae:a8:00, device=bridge, linkActive=true, type=interface, filter=vdsm-no-mac-spoofing, network=ovirtmgmt, deviceId=d00434c2-b72b-4572-9f28-162bfe749013}, {index=0, model=virtio-scsi, specParams={}, device=scsi, type=controller, deviceId=66902ac6-2baf-478d-821c-71f53156d266}],vmName=testtesttest,cpuType=Westmere,fileTransferEnable=true >2014-07-25 13:34:39,055 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVmFromCloudInitVDSCommand] (org.ovirt.thread.pool-8-thread-41) [4f3adf5f] FINISH, CreateVmFromCloudInitVDSCommand, log id: 4df366f >2014-07-25 13:34:39,064 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (org.ovirt.thread.pool-8-thread-41) [4f3adf5f] FINISH, CreateVmVDSCommand, return: WaitForLaunch, log id: 32bc339c >2014-07-25 13:34:39,065 INFO [org.ovirt.engine.core.bll.RunVmCommand] (org.ovirt.thread.pool-8-thread-41) [4f3adf5f] Lock freed to object EngineLock [exclusiveLocks= key: 78722009-2727-4715-91bf-dbbb33ce111e value: VM >, sharedLocks= ] >2014-07-25 13:34:39,072 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-41) [4f3adf5f] Correlation ID: 4f3adf5f, Job ID: 015cfb40-a61b-43a0-9793-637c0531876a, Call Stack: null, Custom Event ID: -1, Message: VM testtesttest was started by admin (Host: rack). >2014-07-25 13:34:41,386 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-78) VM testtesttest 78722009-2727-4715-91bf-dbbb33ce111e moved from WaitForLaunch --> PoweringUp >2014-07-25 13:34:41,388 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVdsCommand] (DefaultQuartzScheduler_Worker-78) START, FullListVdsCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b], vmIds=[78722009-2727-4715-91bf-dbbb33ce111e]), log id: 5e2e6401 >2014-07-25 13:34:41,398 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVdsCommand] (DefaultQuartzScheduler_Worker-78) FINISH, FullListVdsCommand, return: [{displaySecurePort=5901, kvmEnable=true, nicModel=rtl8139,pv, keyboardLayout=en-us, displayIp=10.34.62.208, pauseCode=NOERR, nice=0, displayNetwork=ovirtmgmt, copyPasteEnable=true, timeOffset=0, transparentHugePages=true, vmId=78722009-2727-4715-91bf-dbbb33ce111e, acpiEnable=true, custom={}, spiceSslCipherSuite=DEFAULT, memSize=5120, smp=1, displayPort=5900, emulatedMachine=rhel6.5.0, vmType=kvm, status=Up, memGuaranteedSize=5120, display=qxl, pid=22150, smartcardEnable=false, bootMenuEnable=false, spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,susbredir,ssmartcard, numaTune={mode=preferred}, smpCoresPerSocket=1, maxVCpus=160, clientIp=, devices=[Ljava.lang.Object;@1769b96b, vmName=testtesttest, fileTransferEnable=true, cpuType=Westmere}], log id: 5e2e6401 >2014-07-25 13:34:41,411 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-78) Received a memballoon Device without an address when processing VM 78722009-2727-4715-91bf-dbbb33ce111e devices, skipping device: {alias=balloon0, specParams={model=none}, device=memballoon, type=balloon} >2014-07-25 13:34:41,413 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-78) Received a spice Device without an address when processing VM 78722009-2727-4715-91bf-dbbb33ce111e devices, skipping device: {port=5900, specParams={spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,susbredir,ssmartcard, keyMap=en-us, displayNetwork=ovirtmgmt, copyPasteEnable=true, displayIp=10.34.62.208}, device=spice, tlsPort=5901, type=graphics} >2014-07-25 13:34:43,248 INFO [org.ovirt.engine.core.bll.SetVmTicketCommand] (ajp--127.0.0.1-8702-2) [7314faa8] Running command: SetVmTicketCommand internal: false. Entities affected : ID: 78722009-2727-4715-91bf-dbbb33ce111e Type: VM >2014-07-25 13:34:43,256 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetVmTicketVDSCommand] (ajp--127.0.0.1-8702-2) [7314faa8] START, SetVmTicketVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vmId=78722009-2727-4715-91bf-dbbb33ce111e, ticket=Z9uKew2SVSyz, validTime=120,m userName=admin, userId=fdfc627c-d875-11e0-90f0-83df133b58cc), log id: 76a8d70 >2014-07-25 13:34:43,307 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetVmTicketVDSCommand] (ajp--127.0.0.1-8702-2) [7314faa8] FINISH, SetVmTicketVDSCommand, log id: 76a8d70 >2014-07-25 13:34:43,320 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-2) [7314faa8] Correlation ID: 7314faa8, Call Stack: null, Custom Event ID: -1, Message: user admin initiated console session for VM testtesttest >2014-07-25 13:34:51,728 INFO [org.ovirt.engine.core.bll.SetVmTicketCommand] (ajp--127.0.0.1-8702-5) [8da3838] Running command: SetVmTicketCommand internal: false. Entities affected : ID: 78722009-2727-4715-91bf-dbbb33ce111e Type: VM >2014-07-25 13:34:51,734 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetVmTicketVDSCommand] (ajp--127.0.0.1-8702-5) [8da3838] START, SetVmTicketVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vmId=78722009-2727-4715-91bf-dbbb33ce111e, ticket=WUUWB6jV/ajY, validTime=120,m userName=admin, userId=fdfc627c-d875-11e0-90f0-83df133b58cc), log id: 1149e50c >2014-07-25 13:34:51,775 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetVmTicketVDSCommand] (ajp--127.0.0.1-8702-5) [8da3838] FINISH, SetVmTicketVDSCommand, log id: 1149e50c >2014-07-25 13:34:51,785 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-5) [8da3838] Correlation ID: 8da3838, Call Stack: null, Custom Event ID: -1, Message: user admin initiated console session for VM testtesttest >2014-07-25 13:34:56,381 INFO [org.ovirt.engine.core.bll.SetVmTicketCommand] (ajp--127.0.0.1-8702-8) [7f03b371] Running command: SetVmTicketCommand internal: false. Entities affected : ID: 78722009-2727-4715-91bf-dbbb33ce111e Type: VM >2014-07-25 13:34:56,388 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetVmTicketVDSCommand] (ajp--127.0.0.1-8702-8) [7f03b371] START, SetVmTicketVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vmId=78722009-2727-4715-91bf-dbbb33ce111e, ticket=gdVxAfkMKBLi, validTime=120,m userName=admin, userId=fdfc627c-d875-11e0-90f0-83df133b58cc), log id: 6a44ef1a >2014-07-25 13:34:56,424 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetVmTicketVDSCommand] (ajp--127.0.0.1-8702-8) [7f03b371] FINISH, SetVmTicketVDSCommand, log id: 6a44ef1a >2014-07-25 13:34:56,435 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-8) [7f03b371] Correlation ID: 7f03b371, Call Stack: null, Custom Event ID: -1, Message: user admin initiated console session for VM testtesttest >2014-07-25 13:35:01,904 INFO [org.ovirt.engine.core.bll.SetVmTicketCommand] (ajp--127.0.0.1-8702-6) [5f068865] Running command: SetVmTicketCommand internal: false. Entities affected : ID: 78722009-2727-4715-91bf-dbbb33ce111e Type: VM >2014-07-25 13:35:01,909 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetVmTicketVDSCommand] (ajp--127.0.0.1-8702-6) [5f068865] START, SetVmTicketVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vmId=78722009-2727-4715-91bf-dbbb33ce111e, ticket=OzD3NSenMHYU, validTime=120,m userName=admin, userId=fdfc627c-d875-11e0-90f0-83df133b58cc), log id: 5ae021e2 >2014-07-25 13:35:01,958 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetVmTicketVDSCommand] (ajp--127.0.0.1-8702-6) [5f068865] FINISH, SetVmTicketVDSCommand, log id: 5ae021e2 >2014-07-25 13:35:01,968 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) [5f068865] Correlation ID: 5f068865, Call Stack: null, Custom Event ID: -1, Message: user admin initiated console session for VM testtesttest >2014-07-25 13:35:03,925 INFO [org.ovirt.engine.core.bll.SetVmTicketCommand] (ajp--127.0.0.1-8702-1) [136ab59] Running command: SetVmTicketCommand internal: false. Entities affected : ID: 78722009-2727-4715-91bf-dbbb33ce111e Type: VM >2014-07-25 13:35:03,937 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetVmTicketVDSCommand] (ajp--127.0.0.1-8702-1) [136ab59] START, SetVmTicketVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vmId=78722009-2727-4715-91bf-dbbb33ce111e, ticket=wZzwEsPwNpgh, validTime=120,m userName=admin, userId=fdfc627c-d875-11e0-90f0-83df133b58cc), log id: 2ae4a6b5 >2014-07-25 13:35:03,984 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetVmTicketVDSCommand] (ajp--127.0.0.1-8702-1) [136ab59] FINISH, SetVmTicketVDSCommand, log id: 2ae4a6b5 >2014-07-25 13:35:03,996 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-1) [136ab59] Correlation ID: 136ab59, Call Stack: null, Custom Event ID: -1, Message: user admin initiated console session for VM testtesttest >2014-07-25 13:35:23,004 INFO [org.ovirt.engine.core.bll.SetVmTicketCommand] (ajp--127.0.0.1-8702-9) [32d0d16b] Running command: SetVmTicketCommand internal: false. Entities affected : ID: 78722009-2727-4715-91bf-dbbb33ce111e Type: VM >2014-07-25 13:35:23,009 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetVmTicketVDSCommand] (ajp--127.0.0.1-8702-9) [32d0d16b] START, SetVmTicketVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vmId=78722009-2727-4715-91bf-dbbb33ce111e, ticket=kq2FkK7R8E9w, validTime=120,m userName=admin, userId=fdfc627c-d875-11e0-90f0-83df133b58cc), log id: 3cfbc3f7 >2014-07-25 13:35:23,054 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetVmTicketVDSCommand] (ajp--127.0.0.1-8702-9) [32d0d16b] FINISH, SetVmTicketVDSCommand, log id: 3cfbc3f7 >2014-07-25 13:35:23,064 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-9) [32d0d16b] Correlation ID: 32d0d16b, Call Stack: null, Custom Event ID: -1, Message: user admin initiated console session for VM testtesttest >2014-07-25 13:35:26,815 INFO [org.ovirt.engine.core.bll.SetVmTicketCommand] (ajp--127.0.0.1-8702-8) [fd5ef46] Running command: SetVmTicketCommand internal: false. Entities affected : ID: 78722009-2727-4715-91bf-dbbb33ce111e Type: VM >2014-07-25 13:35:26,825 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetVmTicketVDSCommand] (ajp--127.0.0.1-8702-8) [fd5ef46] START, SetVmTicketVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vmId=78722009-2727-4715-91bf-dbbb33ce111e, ticket=h9B8Ex/9G2vq, validTime=120,m userName=admin, userId=fdfc627c-d875-11e0-90f0-83df133b58cc), log id: 1c20ceae >2014-07-25 13:35:26,869 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetVmTicketVDSCommand] (ajp--127.0.0.1-8702-8) [fd5ef46] FINISH, SetVmTicketVDSCommand, log id: 1c20ceae >2014-07-25 13:35:26,884 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-8) [fd5ef46] Correlation ID: fd5ef46, Call Stack: null, Custom Event ID: -1, Message: user admin initiated console session for VM testtesttest >2014-07-25 13:35:39,207 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-5) VM testtesttest 78722009-2727-4715-91bf-dbbb33ce111e moved from PoweringUp --> Up >2014-07-25 13:35:39,222 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-5) Correlation ID: 4f3adf5f, Job ID: 015cfb40-a61b-43a0-9793-637c0531876a, Call Stack: null, Custom Event ID: -1, Message: VM testtesttest started on Host rack >2014-07-25 13:35:54,288 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (DefaultQuartzScheduler_Worker-40) Setting new tasks map. The map contains now 0 tasks >2014-07-25 13:35:54,289 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (DefaultQuartzScheduler_Worker-40) Cleared all tasks of pool 00000002-0002-0002-0002-0000000001d9. >2014-07-25 13:35:59,997 INFO [org.ovirt.engine.core.bll.SetVmTicketCommand] (ajp--127.0.0.1-8702-6) [6426061] Running command: SetVmTicketCommand internal: false. Entities affected : ID: 78722009-2727-4715-91bf-dbbb33ce111e Type: VM >2014-07-25 13:36:00,003 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetVmTicketVDSCommand] (ajp--127.0.0.1-8702-6) [6426061] START, SetVmTicketVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vmId=78722009-2727-4715-91bf-dbbb33ce111e, ticket=ZVhSbBuWzn64, validTime=120,m userName=admin, userId=fdfc627c-d875-11e0-90f0-83df133b58cc), log id: 7d89e0f3 >2014-07-25 13:36:00,044 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetVmTicketVDSCommand] (ajp--127.0.0.1-8702-6) [6426061] FINISH, SetVmTicketVDSCommand, log id: 7d89e0f3 >2014-07-25 13:36:00,058 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) [6426061] Correlation ID: 6426061, Call Stack: null, Custom Event ID: -1, Message: user admin initiated console session for VM testtesttest >2014-07-25 13:36:18,367 INFO [org.ovirt.engine.core.bll.StopVmCommand] (org.ovirt.thread.pool-8-thread-44) [3149e2e5] Running command: StopVmCommand internal: false. Entities affected : ID: 78722009-2727-4715-91bf-dbbb33ce111e Type: VM >2014-07-25 13:36:18,371 INFO [org.ovirt.engine.core.vdsbroker.DestroyVmVDSCommand] (org.ovirt.thread.pool-8-thread-44) [3149e2e5] START, DestroyVmVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vmId=78722009-2727-4715-91bf-dbbb33ce111e, force=false, secondsToWait=0, gracefully=false, reason=), log id: 428a7c40 >2014-07-25 13:36:18,378 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (org.ovirt.thread.pool-8-thread-44) [3149e2e5] START, DestroyVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vmId=78722009-2727-4715-91bf-dbbb33ce111e, force=false, secondsToWait=0, gracefully=false, reason=), log id: 44ca626f >2014-07-25 13:36:18,803 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (org.ovirt.thread.pool-8-thread-44) [3149e2e5] FINISH, DestroyVDSCommand, log id: 44ca626f >2014-07-25 13:36:18,811 INFO [org.ovirt.engine.core.vdsbroker.DestroyVmVDSCommand] (org.ovirt.thread.pool-8-thread-44) [3149e2e5] FINISH, DestroyVmVDSCommand, return: Down, log id: 428a7c40 >2014-07-25 13:36:18,825 INFO [org.ovirt.engine.core.bll.ProcessDownVmCommand] (org.ovirt.thread.pool-8-thread-49) [783724f9] Running command: ProcessDownVmCommand internal: true. >2014-07-25 13:36:18,834 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-44) [3149e2e5] Correlation ID: 3149e2e5, Job ID: 1a89d746-1ce6-486c-b832-eb3fae4abfd3, Call Stack: null, Custom Event ID: -1, Message: VM testtesttest powered off by admin (Host: rack) (Reason: Not Specified). >2014-07-25 13:36:21,033 INFO [org.ovirt.engine.core.bll.RunVmCommand] (ajp--127.0.0.1-8702-10) [2b8e1554] Lock Acquired to object EngineLock [exclusiveLocks= key: 78722009-2727-4715-91bf-dbbb33ce111e value: VM >, sharedLocks= ] >2014-07-25 13:36:21,115 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (ajp--127.0.0.1-8702-10) [2b8e1554] START, IsVmDuringInitiatingVDSCommand( vmId = 78722009-2727-4715-91bf-dbbb33ce111e), log id: 118a0d1f >2014-07-25 13:36:21,117 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (ajp--127.0.0.1-8702-10) [2b8e1554] FINISH, IsVmDuringInitiatingVDSCommand, return: false, log id: 118a0d1f >2014-07-25 13:36:21,156 INFO [org.ovirt.engine.core.bll.RunVmCommand] (org.ovirt.thread.pool-8-thread-35) [2b8e1554] Running command: RunVmCommand internal: false. Entities affected : ID: 78722009-2727-4715-91bf-dbbb33ce111e Type: VM >2014-07-25 13:36:21,193 INFO [org.ovirt.engine.core.vdsbroker.UpdateVmDynamicDataVDSCommand] (org.ovirt.thread.pool-8-thread-35) [2b8e1554] START, UpdateVmDynamicDataVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vmDynamic=org.ovirt.engine.core.common.businessentities.VmDynamic@4dd29a5c), log id: 12e298f8 >2014-07-25 13:36:21,197 INFO [org.ovirt.engine.core.vdsbroker.UpdateVmDynamicDataVDSCommand] (org.ovirt.thread.pool-8-thread-35) [2b8e1554] FINISH, UpdateVmDynamicDataVDSCommand, log id: 12e298f8 >2014-07-25 13:36:21,203 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (org.ovirt.thread.pool-8-thread-35) [2b8e1554] START, CreateVmVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vmId=78722009-2727-4715-91bf-dbbb33ce111e, vm=VM [testtesttest]), log id: 5c4e97ea >2014-07-25 13:36:21,208 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (org.ovirt.thread.pool-8-thread-35) [2b8e1554] START, CreateVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vmId=78722009-2727-4715-91bf-dbbb33ce111e, vm=VM [testtesttest]), log id: 6c32b26a >2014-07-25 13:36:21,230 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (org.ovirt.thread.pool-8-thread-35) [2b8e1554] org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand kvmEnable=true,keyboardLayout=en-us,nice=0,displayNetwork=ovirtmgmt,copyPasteEnable=true,timeOffset=0,transparentHugePages=true,vmId=78722009-2727-4715-91bf-dbbb33ce111e,acpiEnable=true,custom={},spiceSslCipherSuite=DEFAULT,memSize=5120,smp=1,emulatedMachine=rhel6.5.0,vmType=kvm,memGuaranteedSize=5120,display=qxl,smartcardEnable=false,bootMenuEnable=false,spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,susbredir,ssmartcard,smpCoresPerSocket=1,numaTune={mode=preferred},maxVCpus=160,devices=[{address={bus=0x00, domain=0x0000, slot=0x02, type=pci, function=0x0}, specParams={vram=32768, heads=1}, device=qxl, type=video, deviceId=d962101d-abb7-4fbb-b249-2abdf8b3ee46}, {shared=false, iface=ide, index=2, address={unit=0, bus=1, target=0, controller=0, type=drive}, specParams={path=}, path=, device=cdrom, type=disk, readonly=true, deviceId=a0db1b5f-d569-4b3f-a0e3-3c2992216d80}, {shared=false, index=0, volumeID=b05d9fa4-2cf0-414e-b031-f89fcdf2cf0b, propagateErrors=off, format=raw, type=disk, iface=virtio, address={bus=0x00, domain=0x0000, slot=0x06, type=pci, function=0x0}, domainID=71f82fcd-4fa6-4f56-b228-f4fa4bd4f752, imageID=6cae69b6-2d4f-4e61-993c-dce113dd19e5, specParams={}, optional=false, device=disk, poolID=00000002-0002-0002-0002-0000000001d9, readonly=false, deviceId=6cae69b6-2d4f-4e61-993c-dce113dd19e5}, {nicModel=pv, address={bus=0x00, domain=0x0000, slot=0x03, type=pci, function=0x0}, specParams={outbound={}, inbound={}}, macAddr=00:1a:4a:ae:a8:00, device=bridge, linkActive=true, type=interface, filter=vdsm-no-mac-spoofing, network=ovirtmgmt, deviceId=d00434c2-b72b-4572-9f28-162bfe749013}, {index=0, model=virtio-scsi, address={bus=0x00, domain=0x0000, slot=0x04, type=pci, function=0x0}, specParams={}, device=scsi, type=controller, deviceId=66902ac6-2baf-478d-821c-71f53156d266}],vmName=testtesttest,cpuType=Westmere,fileTransferEnable=true >2014-07-25 13:36:21,265 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (org.ovirt.thread.pool-8-thread-35) [2b8e1554] FINISH, CreateVDSCommand, log id: 6c32b26a >2014-07-25 13:36:21,271 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (org.ovirt.thread.pool-8-thread-35) [2b8e1554] FINISH, CreateVmVDSCommand, return: WaitForLaunch, log id: 5c4e97ea >2014-07-25 13:36:21,272 INFO [org.ovirt.engine.core.bll.RunVmCommand] (org.ovirt.thread.pool-8-thread-35) [2b8e1554] Lock freed to object EngineLock [exclusiveLocks= key: 78722009-2727-4715-91bf-dbbb33ce111e value: VM >, sharedLocks= ] >2014-07-25 13:36:21,278 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-35) [2b8e1554] Correlation ID: 2b8e1554, Job ID: 3cbd46e7-5428-4259-90cf-b4663f92e4bc, Call Stack: null, Custom Event ID: -1, Message: VM testtesttest was started by admin (Host: rack). >2014-07-25 13:36:24,901 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-8) VM testtesttest 78722009-2727-4715-91bf-dbbb33ce111e moved from WaitForLaunch --> PoweringUp >2014-07-25 13:36:24,904 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVdsCommand] (DefaultQuartzScheduler_Worker-8) START, FullListVdsCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b], vmIds=[78722009-2727-4715-91bf-dbbb33ce111e]), log id: 4da758dd >2014-07-25 13:36:24,915 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVdsCommand] (DefaultQuartzScheduler_Worker-8) FINISH, FullListVdsCommand, return: [{displaySecurePort=5901, kvmEnable=true, nicModel=rtl8139,pv, keyboardLayout=en-us, displayIp=10.34.62.208, pauseCode=NOERR, nice=0, displayNetwork=ovirtmgmt, copyPasteEnable=true, timeOffset=0, transparentHugePages=true, vmId=78722009-2727-4715-91bf-dbbb33ce111e, acpiEnable=true, custom={}, spiceSslCipherSuite=DEFAULT, memSize=5120, smp=1, displayPort=5900, emulatedMachine=rhel6.5.0, vmType=kvm, status=Up, memGuaranteedSize=5120, display=qxl, pid=22477, smartcardEnable=false, bootMenuEnable=false, spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,susbredir,ssmartcard, numaTune={mode=preferred}, smpCoresPerSocket=1, maxVCpus=160, clientIp=, devices=[Ljava.lang.Object;@25bb3cff, vmName=testtesttest, fileTransferEnable=true, cpuType=Westmere}], log id: 4da758dd >2014-07-25 13:36:24,921 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-8) Received a memballoon Device without an address when processing VM 78722009-2727-4715-91bf-dbbb33ce111e devices, skipping device: {alias=balloon0, specParams={model=none}, device=memballoon, type=balloon} >2014-07-25 13:36:24,923 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-8) Received a spice Device without an address when processing VM 78722009-2727-4715-91bf-dbbb33ce111e devices, skipping device: {port=5900, specParams={spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,susbredir,ssmartcard, keyMap=en-us, displayNetwork=ovirtmgmt, copyPasteEnable=true, displayIp=10.34.62.208}, device=spice, tlsPort=5901, type=graphics} >2014-07-25 13:36:26,878 INFO [org.ovirt.engine.core.bll.SetVmTicketCommand] (ajp--127.0.0.1-8702-7) [4dbf49a2] Running command: SetVmTicketCommand internal: false. Entities affected : ID: 78722009-2727-4715-91bf-dbbb33ce111e Type: VM >2014-07-25 13:36:26,884 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetVmTicketVDSCommand] (ajp--127.0.0.1-8702-7) [4dbf49a2] START, SetVmTicketVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vmId=78722009-2727-4715-91bf-dbbb33ce111e, ticket=dYXS3cMpK8FZ, validTime=120,m userName=admin, userId=fdfc627c-d875-11e0-90f0-83df133b58cc), log id: 7bd1bc81 >2014-07-25 13:36:27,021 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetVmTicketVDSCommand] (ajp--127.0.0.1-8702-7) [4dbf49a2] FINISH, SetVmTicketVDSCommand, log id: 7bd1bc81 >2014-07-25 13:36:27,033 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-7) [4dbf49a2] Correlation ID: 4dbf49a2, Call Stack: null, Custom Event ID: -1, Message: user admin initiated console session for VM testtesttest >2014-07-25 13:36:31,070 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-25) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin is connected to VM testtesttest. >2014-07-25 13:37:22,671 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-43) VM testtesttest 78722009-2727-4715-91bf-dbbb33ce111e moved from PoweringUp --> Up >2014-07-25 13:37:22,686 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-43) Correlation ID: 2b8e1554, Job ID: 3cbd46e7-5428-4259-90cf-b4663f92e4bc, Call Stack: null, Custom Event ID: -1, Message: VM testtesttest started on Host rack >2014-07-25 13:46:36,595 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-88) VM testtesttest 78722009-2727-4715-91bf-dbbb33ce111e moved from Up --> RebootInProgress >2014-07-25 13:46:36,598 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVdsCommand] (DefaultQuartzScheduler_Worker-88) START, FullListVdsCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b], vmIds=[78722009-2727-4715-91bf-dbbb33ce111e]), log id: 6b7c285 >2014-07-25 13:46:36,609 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVdsCommand] (DefaultQuartzScheduler_Worker-88) FINISH, FullListVdsCommand, return: [{displaySecurePort=5901, kvmEnable=true, nicModel=rtl8139,pv, keyboardLayout=en-us, displayIp=10.34.62.208, pauseCode=NOERR, nice=0, displayNetwork=ovirtmgmt, copyPasteEnable=true, timeOffset=0, transparentHugePages=true, vmId=78722009-2727-4715-91bf-dbbb33ce111e, acpiEnable=true, custom={}, spiceSslCipherSuite=DEFAULT, memSize=5120, smp=1, displayPort=5900, emulatedMachine=rhel6.5.0, vmType=kvm, status=Up, memGuaranteedSize=5120, display=qxl, pid=22477, smartcardEnable=false, bootMenuEnable=false, spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,susbredir,ssmartcard, numaTune={mode=preferred}, smpCoresPerSocket=1, maxVCpus=160, clientIp=10.34.130.128, devices=[Ljava.lang.Object;@a43ad35, vmName=testtesttest, fileTransferEnable=true, cpuType=Westmere}], log id: 6b7c285 >2014-07-25 13:46:36,619 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-88) Received a memballoon Device without an address when processing VM 78722009-2727-4715-91bf-dbbb33ce111e devices, skipping device: {alias=balloon0, specParams={model=none}, device=memballoon, type=balloon} >2014-07-25 13:46:36,621 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-88) Received a spice Device without an address when processing VM 78722009-2727-4715-91bf-dbbb33ce111e devices, skipping device: {port=5900, specParams={spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,susbredir,ssmartcard, keyMap=en-us, displayNetwork=ovirtmgmt, copyPasteEnable=true, displayIp=10.34.62.208}, device=spice, tlsPort=5901, type=graphics} >2014-07-25 13:47:34,316 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-16) VM testtesttest 78722009-2727-4715-91bf-dbbb33ce111e moved from RebootInProgress --> Up >2014-07-25 14:09:34,257 WARN [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-8-thread-21) Executing a command: java.util.concurrent.FutureTask , but note that there are 0 tasks in the queue. >2014-07-25 14:23:54,290 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-40) Principal admin::internal synchronized >2014-07-25 14:23:55,778 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-58) Attempting to update VMs/Templates Ovf. >2014-07-25 14:23:55,908 INFO [org.ovirt.engine.core.bll.ProcessOvfUpdateForStorageDomainCommand] (DefaultQuartzScheduler_Worker-58) [52e4a03c] Lock Acquired to object EngineLock [exclusiveLocks= key: 07c061c2-77f4-4c02-8402-e4dff773b216 value: DISK >key: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 value: STORAGE >key: 90108fe1-56a4-4f0e-b301-fa0ef46ba717 value: DISK >, sharedLocks= key: 00000002-0002-0002-0002-0000000001d9 value: OVF_UPDATE >] >2014-07-25 14:23:55,912 INFO [org.ovirt.engine.core.bll.ProcessOvfUpdateForStorageDomainCommand] (DefaultQuartzScheduler_Worker-58) [52e4a03c] Running command: ProcessOvfUpdateForStorageDomainCommand internal: true. Entities affected : ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage >2014-07-25 14:23:55,943 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (DefaultQuartzScheduler_Worker-58) [52e4a03c] START, SetVolumeDescriptionVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, storageDomainId = 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752, imageGroupId = 90108fe1-56a4-4f0e-b301-fa0ef46ba717, imageId = 8596469e-37df-4d9e-9793-d238238131f5), log id: 57d4eb14 >2014-07-25 14:23:55,945 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (DefaultQuartzScheduler_Worker-58) [52e4a03c] -- executeIrsBrokerCommand: calling 'setVolumeDescription' >2014-07-25 14:23:55,947 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (DefaultQuartzScheduler_Worker-58) [52e4a03c] -- setVolumeDescription parameters: > spUUID=00000002-0002-0002-0002-0000000001d9 > sdUUID=71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 > imageGroupGUID=90108fe1-56a4-4f0e-b301-fa0ef46ba717 > volUUID=8596469e-37df-4d9e-9793-d238238131f5 > description={"Updated":false,"Disk Description":"OVF_STORE","Storage Domains":[{"uuid":"71f82fcd-4fa6-4f56-b228-f4fa4bd4f752"}],"Last Updated":"Fri Jul 25 13:24:08 CEST 2014"} > >2014-07-25 14:23:56,070 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (DefaultQuartzScheduler_Worker-58) [52e4a03c] FINISH, SetVolumeDescriptionVDSCommand, log id: 57d4eb14 >2014-07-25 14:23:56,099 INFO [org.ovirt.engine.core.bll.UploadStreamCommand] (DefaultQuartzScheduler_Worker-58) [5e092991] Running command: UploadStreamCommand internal: true. Entities affected : ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage >2014-07-25 14:23:56,104 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.UploadStreamVDSCommand] (DefaultQuartzScheduler_Worker-58) [5e092991] START, UploadStreamVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 7d9ee577 >2014-07-25 14:23:56,108 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.UploadStreamVDSCommand] (DefaultQuartzScheduler_Worker-58) [5e092991] -- executeVdsBrokerCommand: >2014-07-25 14:23:56,111 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.UploadStreamVDSCommand] (DefaultQuartzScheduler_Worker-58) [5e092991] -- parameters: > spUUID=00000002-0002-0002-0002-0000000001d9 > sdUUID=71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 > imageGUID=90108fe1-56a4-4f0e-b301-fa0ef46ba717 > volUUID=8596469e-37df-4d9e-9793-d238238131f5 > size=10,240 > >2014-07-25 14:23:57,645 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.UploadStreamVDSCommand] (DefaultQuartzScheduler_Worker-58) [5e092991] FINISH, UploadStreamVDSCommand, log id: 7d9ee577 >2014-07-25 14:23:57,649 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (DefaultQuartzScheduler_Worker-58) [5e092991] CommandAsyncTask::Adding CommandMultiAsyncTasks object for command 09ff096c-0935-490c-b787-df4037d31160 >2014-07-25 14:23:57,650 INFO [org.ovirt.engine.core.bll.CommandMultiAsyncTasks] (DefaultQuartzScheduler_Worker-58) [5e092991] CommandMultiAsyncTasks::AttachTask: Attaching task e59dd314-6a4d-43c0-b873-8403d1791fb9 to command 09ff096c-0935-490c-b787-df4037d31160. >2014-07-25 14:23:57,657 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (DefaultQuartzScheduler_Worker-58) [5e092991] Adding task e59dd314-6a4d-43c0-b873-8403d1791fb9 (Parent Command ProcessOvfUpdateForStorageDomain, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters), polling hasn't started yet.. >2014-07-25 14:23:57,658 INFO [org.ovirt.engine.core.bll.UploadStreamCommand] (DefaultQuartzScheduler_Worker-58) [5e092991] Lock freed to object EngineLock [exclusiveLocks= key: 07c061c2-77f4-4c02-8402-e4dff773b216 value: DISK >key: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 value: STORAGE >key: 90108fe1-56a4-4f0e-b301-fa0ef46ba717 value: DISK >, sharedLocks= key: 00000002-0002-0002-0002-0000000001d9 value: OVF_UPDATE >] >2014-07-25 14:23:57,664 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (DefaultQuartzScheduler_Worker-58) [5e092991] START, SetVolumeDescriptionVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, storageDomainId = 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752, imageGroupId = 90108fe1-56a4-4f0e-b301-fa0ef46ba717, imageId = 8596469e-37df-4d9e-9793-d238238131f5), log id: 707eb765 >2014-07-25 14:23:57,666 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (DefaultQuartzScheduler_Worker-58) [5e092991] -- executeIrsBrokerCommand: calling 'setVolumeDescription' >2014-07-25 14:23:57,667 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (DefaultQuartzScheduler_Worker-58) [5e092991] -- setVolumeDescription parameters: > spUUID=00000002-0002-0002-0002-0000000001d9 > sdUUID=71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 > imageGroupGUID=90108fe1-56a4-4f0e-b301-fa0ef46ba717 > volUUID=8596469e-37df-4d9e-9793-d238238131f5 > description={"Updated":true,"Disk Description":"OVF_STORE","Storage Domains":[{"uuid":"71f82fcd-4fa6-4f56-b228-f4fa4bd4f752"}],"Last Updated":"Fri Jul 25 14:23:55 CEST 2014","Size":10240} > >2014-07-25 14:23:57,823 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (DefaultQuartzScheduler_Worker-58) [5e092991] FINISH, SetVolumeDescriptionVDSCommand, log id: 707eb765 >2014-07-25 14:23:57,828 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (DefaultQuartzScheduler_Worker-58) [5e092991] START, SetVolumeDescriptionVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, storageDomainId = 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752, imageGroupId = 07c061c2-77f4-4c02-8402-e4dff773b216, imageId = 1f5f82d1-85c2-4386-8764-02bab6dcbc21), log id: 25645b8c >2014-07-25 14:23:57,830 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (DefaultQuartzScheduler_Worker-58) [5e092991] -- executeIrsBrokerCommand: calling 'setVolumeDescription' >2014-07-25 14:23:57,831 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (DefaultQuartzScheduler_Worker-58) [5e092991] -- setVolumeDescription parameters: > spUUID=00000002-0002-0002-0002-0000000001d9 > sdUUID=71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 > imageGroupGUID=07c061c2-77f4-4c02-8402-e4dff773b216 > volUUID=1f5f82d1-85c2-4386-8764-02bab6dcbc21 > description={"Updated":false,"Disk Description":"OVF_STORE","Storage Domains":[{"uuid":"71f82fcd-4fa6-4f56-b228-f4fa4bd4f752"}],"Last Updated":"Fri Jul 25 13:24:08 CEST 2014"} > >2014-07-25 14:23:58,039 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (DefaultQuartzScheduler_Worker-58) [5e092991] FINISH, SetVolumeDescriptionVDSCommand, log id: 25645b8c >2014-07-25 14:23:58,080 INFO [org.ovirt.engine.core.bll.UploadStreamCommand] (DefaultQuartzScheduler_Worker-58) [266916e1] Running command: UploadStreamCommand internal: true. Entities affected : ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage >2014-07-25 14:23:58,083 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.UploadStreamVDSCommand] (DefaultQuartzScheduler_Worker-58) [266916e1] START, UploadStreamVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 16e0ce98 >2014-07-25 14:23:58,084 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.UploadStreamVDSCommand] (DefaultQuartzScheduler_Worker-58) [266916e1] -- executeVdsBrokerCommand: >2014-07-25 14:23:58,085 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.UploadStreamVDSCommand] (DefaultQuartzScheduler_Worker-58) [266916e1] -- parameters: > spUUID=00000002-0002-0002-0002-0000000001d9 > sdUUID=71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 > imageGUID=07c061c2-77f4-4c02-8402-e4dff773b216 > volUUID=1f5f82d1-85c2-4386-8764-02bab6dcbc21 > size=10,240 > >2014-07-25 14:23:58,873 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.UploadStreamVDSCommand] (DefaultQuartzScheduler_Worker-58) [266916e1] FINISH, UploadStreamVDSCommand, log id: 16e0ce98 >2014-07-25 14:23:58,875 INFO [org.ovirt.engine.core.bll.CommandMultiAsyncTasks] (DefaultQuartzScheduler_Worker-58) [266916e1] CommandMultiAsyncTasks::AttachTask: Attaching task 877232e6-8358-498a-8334-857f5e5748fb to command 09ff096c-0935-490c-b787-df4037d31160. >2014-07-25 14:23:58,882 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (DefaultQuartzScheduler_Worker-58) [266916e1] Adding task 877232e6-8358-498a-8334-857f5e5748fb (Parent Command ProcessOvfUpdateForStorageDomain, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters), polling hasn't started yet.. >2014-07-25 14:23:58,884 WARN [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (DefaultQuartzScheduler_Worker-58) [266916e1] Trying to release a shared lock for key: 00000002-0002-0002-0002-0000000001d9OVF_UPDATE , but lock does not exist >2014-07-25 14:23:58,885 WARN [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (DefaultQuartzScheduler_Worker-58) [266916e1] Trying to release exclusive lock which does not exist, lock key: 07c061c2-77f4-4c02-8402-e4dff773b216DISK >2014-07-25 14:23:58,886 WARN [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (DefaultQuartzScheduler_Worker-58) [266916e1] Trying to release exclusive lock which does not exist, lock key: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752STORAGE >2014-07-25 14:23:58,887 WARN [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (DefaultQuartzScheduler_Worker-58) [266916e1] Trying to release exclusive lock which does not exist, lock key: 90108fe1-56a4-4f0e-b301-fa0ef46ba717DISK >2014-07-25 14:23:58,888 INFO [org.ovirt.engine.core.bll.UploadStreamCommand] (DefaultQuartzScheduler_Worker-58) [266916e1] Lock freed to object EngineLock [exclusiveLocks= key: 07c061c2-77f4-4c02-8402-e4dff773b216 value: DISK >key: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 value: STORAGE >key: 90108fe1-56a4-4f0e-b301-fa0ef46ba717 value: DISK >, sharedLocks= key: 00000002-0002-0002-0002-0000000001d9 value: OVF_UPDATE >] >2014-07-25 14:23:58,891 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (DefaultQuartzScheduler_Worker-58) [266916e1] START, SetVolumeDescriptionVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, storageDomainId = 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752, imageGroupId = 07c061c2-77f4-4c02-8402-e4dff773b216, imageId = 1f5f82d1-85c2-4386-8764-02bab6dcbc21), log id: 46da6ba3 >2014-07-25 14:23:58,893 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (DefaultQuartzScheduler_Worker-58) [266916e1] -- executeIrsBrokerCommand: calling 'setVolumeDescription' >2014-07-25 14:23:58,894 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (DefaultQuartzScheduler_Worker-58) [266916e1] -- setVolumeDescription parameters: > spUUID=00000002-0002-0002-0002-0000000001d9 > sdUUID=71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 > imageGroupGUID=07c061c2-77f4-4c02-8402-e4dff773b216 > volUUID=1f5f82d1-85c2-4386-8764-02bab6dcbc21 > description={"Updated":true,"Disk Description":"OVF_STORE","Storage Domains":[{"uuid":"71f82fcd-4fa6-4f56-b228-f4fa4bd4f752"}],"Last Updated":"Fri Jul 25 14:23:55 CEST 2014","Size":10240} > >2014-07-25 14:23:59,104 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (DefaultQuartzScheduler_Worker-58) [266916e1] FINISH, SetVolumeDescriptionVDSCommand, log id: 46da6ba3 >2014-07-25 14:23:59,107 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (DefaultQuartzScheduler_Worker-58) [266916e1] BaseAsyncTask::startPollingTask: Starting to poll task e59dd314-6a4d-43c0-b873-8403d1791fb9. >2014-07-25 14:23:59,109 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (DefaultQuartzScheduler_Worker-58) [266916e1] BaseAsyncTask::startPollingTask: Starting to poll task 877232e6-8358-498a-8334-857f5e5748fb. >2014-07-25 14:24:08,915 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (DefaultQuartzScheduler_Worker-80) Polling and updating Async Tasks: 2 tasks, 2 tasks to poll now >2014-07-25 14:24:08,922 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (DefaultQuartzScheduler_Worker-80) SPMAsyncTask::PollTask: Polling task e59dd314-6a4d-43c0-b873-8403d1791fb9 (Parent Command ProcessOvfUpdateForStorageDomain, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) returned status finished, result 'success'. >2014-07-25 14:24:08,923 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (DefaultQuartzScheduler_Worker-80) BaseAsyncTask::OnTaskEndSuccess: Task e59dd314-6a4d-43c0-b873-8403d1791fb9 (Parent Command ProcessOvfUpdateForStorageDomain, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) ended successfully. >2014-07-25 14:24:08,926 INFO [org.ovirt.engine.core.bll.CommandMultiAsyncTasks] (DefaultQuartzScheduler_Worker-80) Task with DB Task ID 750b96d9-f392-4006-8f82-aedd720e661f and VDSM Task ID 877232e6-8358-498a-8334-857f5e5748fb is in state Polling. End action for command 09ff096c-0935-490c-b787-df4037d31160 will proceed when all the entitys tasks are completed. >2014-07-25 14:24:08,928 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (DefaultQuartzScheduler_Worker-80) SPMAsyncTask::PollTask: Polling task 877232e6-8358-498a-8334-857f5e5748fb (Parent Command ProcessOvfUpdateForStorageDomain, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) returned status finished, result 'success'. >2014-07-25 14:24:08,930 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (DefaultQuartzScheduler_Worker-80) BaseAsyncTask::OnTaskEndSuccess: Task 877232e6-8358-498a-8334-857f5e5748fb (Parent Command ProcessOvfUpdateForStorageDomain, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) ended successfully. >2014-07-25 14:24:08,932 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (DefaultQuartzScheduler_Worker-80) CommandAsyncTask::EndActionIfNecessary: All tasks of command 09ff096c-0935-490c-b787-df4037d31160 has ended -> executing endAction >2014-07-25 14:24:08,933 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (DefaultQuartzScheduler_Worker-80) CommandAsyncTask::endAction: Ending action for 2 tasks (command ID: 09ff096c-0935-490c-b787-df4037d31160): calling endAction . >2014-07-25 14:24:08,935 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-8-thread-43) CommandAsyncTask::EndCommandAction [within thread] context: Attempting to endAction ProcessOvfUpdateForStorageDomain, executionIndex: 0 >2014-07-25 14:24:08,978 INFO [org.ovirt.engine.core.bll.ProcessOvfUpdateForStorageDomainCommand] (org.ovirt.thread.pool-8-thread-43) [52e4a03c] Ending command successfully: org.ovirt.engine.core.bll.ProcessOvfUpdateForStorageDomainCommand >2014-07-25 14:24:08,979 WARN [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (org.ovirt.thread.pool-8-thread-43) [52e4a03c] Trying to release a shared lock for key: 00000002-0002-0002-0002-0000000001d9OVF_UPDATE , but lock does not exist >2014-07-25 14:24:08,981 WARN [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (org.ovirt.thread.pool-8-thread-43) [52e4a03c] Trying to release exclusive lock which does not exist, lock key: 07c061c2-77f4-4c02-8402-e4dff773b216DISK >2014-07-25 14:24:08,982 WARN [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (org.ovirt.thread.pool-8-thread-43) [52e4a03c] Trying to release exclusive lock which does not exist, lock key: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752STORAGE >2014-07-25 14:24:08,983 WARN [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (org.ovirt.thread.pool-8-thread-43) [52e4a03c] Trying to release exclusive lock which does not exist, lock key: 90108fe1-56a4-4f0e-b301-fa0ef46ba717DISK >2014-07-25 14:24:08,984 INFO [org.ovirt.engine.core.bll.ProcessOvfUpdateForStorageDomainCommand] (org.ovirt.thread.pool-8-thread-43) [52e4a03c] Lock freed to object EngineLock [exclusiveLocks= key: 07c061c2-77f4-4c02-8402-e4dff773b216 value: DISK >key: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 value: STORAGE >key: 90108fe1-56a4-4f0e-b301-fa0ef46ba717 value: DISK >, sharedLocks= key: 00000002-0002-0002-0002-0000000001d9 value: OVF_UPDATE >] >2014-07-25 14:24:08,987 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-8-thread-43) [52e4a03c] CommandAsyncTask::HandleEndActionResult [within thread]: endAction for action type ProcessOvfUpdateForStorageDomain completed, handling the result. >2014-07-25 14:24:08,988 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-8-thread-43) [52e4a03c] CommandAsyncTask::HandleEndActionResult [within thread]: endAction for action type ProcessOvfUpdateForStorageDomain succeeded, clearing tasks. >2014-07-25 14:24:08,990 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (org.ovirt.thread.pool-8-thread-43) [52e4a03c] SPMAsyncTask::ClearAsyncTask: Attempting to clear task 877232e6-8358-498a-8334-857f5e5748fb >2014-07-25 14:24:08,991 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-43) [52e4a03c] START, SPMClearTaskVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, taskId = 877232e6-8358-498a-8334-857f5e5748fb), log id: 412a9eb0 >2014-07-25 14:24:08,995 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-43) [52e4a03c] START, HSMClearTaskVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, taskId=877232e6-8358-498a-8334-857f5e5748fb), log id: 5518563c >2014-07-25 14:24:09,041 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-43) [52e4a03c] FINISH, HSMClearTaskVDSCommand, log id: 5518563c >2014-07-25 14:24:09,042 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-43) [52e4a03c] FINISH, SPMClearTaskVDSCommand, log id: 412a9eb0 >2014-07-25 14:24:09,048 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (org.ovirt.thread.pool-8-thread-43) [52e4a03c] BaseAsyncTask::RemoveTaskFromDB: Removed task 877232e6-8358-498a-8334-857f5e5748fb from DataBase >2014-07-25 14:24:09,049 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (org.ovirt.thread.pool-8-thread-43) [52e4a03c] SPMAsyncTask::ClearAsyncTask: Attempting to clear task e59dd314-6a4d-43c0-b873-8403d1791fb9 >2014-07-25 14:24:09,050 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-43) [52e4a03c] START, SPMClearTaskVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, taskId = e59dd314-6a4d-43c0-b873-8403d1791fb9), log id: 31e1683d >2014-07-25 14:24:09,053 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-43) [52e4a03c] START, HSMClearTaskVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, taskId=e59dd314-6a4d-43c0-b873-8403d1791fb9), log id: 4365f981 >2014-07-25 14:24:09,167 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-43) [52e4a03c] FINISH, HSMClearTaskVDSCommand, log id: 4365f981 >2014-07-25 14:24:09,168 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-43) [52e4a03c] FINISH, SPMClearTaskVDSCommand, log id: 31e1683d >2014-07-25 14:24:09,174 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (org.ovirt.thread.pool-8-thread-43) [52e4a03c] BaseAsyncTask::RemoveTaskFromDB: Removed task e59dd314-6a4d-43c0-b873-8403d1791fb9 from DataBase >2014-07-25 14:24:09,174 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-8-thread-43) [52e4a03c] CommandAsyncTask::HandleEndActionResult [within thread]: Removing CommandMultiAsyncTasks object for entity 09ff096c-0935-490c-b787-df4037d31160 >2014-07-25 14:25:24,331 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (DefaultQuartzScheduler_Worker-47) Setting new tasks map. The map contains now 0 tasks >2014-07-25 14:25:24,332 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (DefaultQuartzScheduler_Worker-47) Cleared all tasks of pool 00000002-0002-0002-0002-0000000001d9. >2014-07-25 15:23:54,297 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-94) Principal admin::internal synchronized >2014-07-25 15:23:59,109 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-15) Attempting to update VMs/Templates Ovf. >2014-07-25 15:37:15,729 WARN [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-8-thread-31) Executing a command: java.util.concurrent.FutureTask , but note that there are 1 tasks in the queue. >2014-07-25 16:13:52,364 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-55) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin got disconnected from VM testtesttest. >2014-07-25 16:23:54,305 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-23) Principal admin::internal synchronized >2014-07-25 16:23:59,142 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-43) Attempting to update VMs/Templates Ovf. >2014-07-25 17:15:38,518 WARN [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-8-thread-50) Executing a command: java.util.concurrent.FutureTask , but note that there are 1 tasks in the queue. >2014-07-25 17:23:54,314 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-73) Principal admin::internal synchronized >2014-07-25 17:23:59,167 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-98) Attempting to update VMs/Templates Ovf. >2014-07-25 18:20:56,511 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-7) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage domain 35 experienced a high latency of 5.11967 seconds from host rack. This may cause performance and functional issues. Please consult your Storage Administrator. >2014-07-25 18:23:54,322 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-99) Principal admin::internal synchronized >2014-07-25 18:23:59,182 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-3) Attempting to update VMs/Templates Ovf. >2014-07-25 19:23:54,328 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-31) Principal admin::internal synchronized >2014-07-25 19:23:59,194 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-45) Attempting to update VMs/Templates Ovf. >2014-07-25 20:23:54,334 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-12) Principal admin::internal synchronized >2014-07-25 20:23:59,224 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-27) Attempting to update VMs/Templates Ovf. >2014-07-25 21:14:53,157 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-11) domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 in problem. vds: rack >2014-07-25 21:16:08,810 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-86) VM testtesttest 78722009-2727-4715-91bf-dbbb33ce111e moved from Up --> NotResponding >2014-07-25 21:16:08,885 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-86) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: VM testtesttest is not responding. >2014-07-25 21:16:18,066 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-28) Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2014-07-25 21:16:18,071 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-28) [8ef1638] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-07-25 21:16:18,076 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-28) [8ef1638] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Active to Unknown, reason : null >2014-07-25 21:16:18,094 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-28) [8ef1638] Correlation ID: 8ef1638, Call Stack: null, Custom Event ID: -1, Message: Invalid status on Data Center Default. Setting Data Center status to Non Responsive (On host rack, Error: Cannot access master storage domain). >2014-07-25 21:18:18,106 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-28) [8ef1638] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2014-07-25 21:18:18,108 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-28) [8ef1638] hostFromVds::selectedVds - rack, spmStatus returned null! >2014-07-25 21:18:18,110 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-28) [8ef1638] IrsBroker::Failed::GetStoragePoolInfoVDS >2014-07-25 21:18:18,111 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-28) [8ef1638] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2014-07-25 21:18:18,181 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-34) [1efdb026] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage >2014-07-25 21:18:18,204 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-34) [1efdb026] START, ResetIrsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, ignoreStopFailed = true), log id: 36c0c0a3 >2014-07-25 21:18:18,213 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-34) [1efdb026] START, SpmStopVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9), log id: 41a9228a >2014-07-25 21:18:18,219 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-34) [1efdb026] SpmStopVDSCommand::Stopping SPM on vds rack, pool id 00000002-0002-0002-0002-0000000001d9 >2014-07-25 21:20:00,010 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-38) Autorecovering 1 storage domains >2014-07-25 21:20:00,012 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-38) Autorecovering storage domains id: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 >2014-07-25 21:20:00,016 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-38) [1b9ef0f] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage >2014-07-25 21:20:00,018 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-38) [1b9ef0f] ConnectDomainToStorage. Before Connect all hosts to pool. Time:7/25/14 9:20 PM >2014-07-25 21:20:00,040 INFO [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-8-thread-19) [49da9dd2] Running command: ConnectStorageToVdsCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2014-07-25 21:20:00,088 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-19) [49da9dd2] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 6c60160d >2014-07-25 21:21:18,087 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-100) Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2014-07-25 21:21:18,090 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-100) hostFromVds::selectedVds - rack, spmStatus returned null! >2014-07-25 21:21:18,091 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-100) IrsBroker::Failed::GetStoragePoolInfoVDS >2014-07-25 21:21:18,092 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-100) Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2014-07-25 21:21:18,223 WARN [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-34) [1efdb026] could not stop spm of pool 00000002-0002-0002-0002-0000000001d9 on vds 17f7fe56-5f32-40ef-b098-611b84484e0b - reason: org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcRunTimeException: java.util.concurrent.TimeoutException >2014-07-25 21:21:18,225 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-34) [1efdb026] FINISH, SpmStopVDSCommand, log id: 41a9228a >2014-07-25 21:21:18,234 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-34) [1efdb026] FINISH, ResetIrsVDSCommand, log id: 36c0c0a3 >2014-07-25 21:21:18,248 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-34) [1efdb026] Correlation ID: 1efdb026, Job ID: 94c9f882-4940-4be9-8248-5261de5619db, Call Stack: null, Custom Event ID: -1, Message: Failed to Reconstruct Master Domain for Data Center Default. >2014-07-25 21:21:18,256 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-42) [8ef1638] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue >2014-07-25 21:21:18,259 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-47) starting processDomainRecovery for domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 >2014-07-25 21:21:18,320 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-47) Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 was reported by all hosts in status UP as problematic. Not moving the domain to NonOperational because it is being reconstructed now. >2014-07-25 21:21:18,366 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-47) [12262c2a] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage >2014-07-25 21:21:18,397 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-47) [12262c2a] Removing vds [] from the domain in maintenance cache >2014-07-25 21:21:18,398 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-47) [12262c2a] Removing host(s) [] from hosts unseen domain report cache >2014-07-25 21:21:18,399 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-42) [8ef1638] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue >2014-07-25 21:22:18,099 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetStatsVDSCommand] (DefaultQuartzScheduler_Worker-32) Command GetStatsVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-07-25 21:22:18,100 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-19) [49da9dd2] Command ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]) execution failed. Exception: VDSNetworkException: org.apache.commons.httpclient.NoHttpResponseException: The server 10.34.62.208 failed to respond >2014-07-25 21:22:18,104 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-19) [49da9dd2] FINISH, ConnectStorageServerVDSCommand, log id: 6c60160d >2014-07-25 21:22:18,103 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-32) Host rack is not responding. It will stay in Connecting state for a grace period of $60 seconds and after that an attempt to fence the host will be issued. >2014-07-25 21:22:18,101 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-25) Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: VDSNetworkException: org.apache.commons.httpclient.NoHttpResponseException: The server 10.34.62.208 failed to respond >2014-07-25 21:22:18,108 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-32) vds::refreshVdsStats Failed getVdsStats, vds = 17f7fe56-5f32-40ef-b098-611b84484e0b : rack, error = VDSNetworkException: java.net.ConnectException: Connection refused >2014-07-25 21:22:18,106 ERROR [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-8-thread-19) [49da9dd2] Command org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: org.apache.commons.httpclient.NoHttpResponseException: The server 10.34.62.208 failed to respond (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-07-25 21:22:18,110 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-25) hostFromVds::selectedVds - rack, spmStatus returned null! >2014-07-25 21:22:18,125 ERROR [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-8-thread-19) [49da9dd2] Transaction rolled-back for command: org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand. >2014-07-25 21:22:18,112 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-32) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsStats(VdsUpdateRunTimeInfo.java:669) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:623) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:235) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) [:1.7.0_55] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_55] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_55] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_55] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_55] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_55] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_55] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_55] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy82.getVdsStats(Unknown Source) > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_55] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_55] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_55] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_55] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_55] > >2014-07-25 21:22:18,126 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-38) [1b9ef0f] ConnectDomainToStorage. After Connect all hosts to pool. Time:7/25/14 9:22 PM >2014-07-25 21:22:18,153 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-32) Failed to refresh VDS , vds = 17f7fe56-5f32-40ef-b098-611b84484e0b : rack, VDS Network Error, continuing. >java.net.ConnectException: Connection refused >2014-07-25 21:22:21,430 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-36) START, GetHardwareInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]), log id: 42662e48 >2014-07-25 21:22:21,481 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-36) FINISH, GetHardwareInfoVDSCommand, log id: 42662e48 >2014-07-25 21:22:21,517 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-36) [3060a9b0] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS >2014-07-25 21:22:21,531 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-36) [3060a9b0] VM testtesttest 78722009-2727-4715-91bf-dbbb33ce111e moved from NotResponding --> Paused >2014-07-25 21:22:21,537 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-36) [e9159fd] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-07-25 21:22:21,556 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-36) [b23fc58] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-07-25 21:22:21,573 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-36) [b23fc58] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, storageType = NFS, connectionList = [{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 195e5835 >2014-07-25 21:22:21,586 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-36) [b23fc58] FINISH, ConnectStorageServerVDSCommand, return: {5083f370-af09-457b-9b73-3440e8b65a07=0}, log id: 195e5835 >2014-07-25 21:22:21,588 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-36) [b23fc58] Host rack storage connection was succeeded >2014-07-25 21:22:21,595 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-12) START, ConnectStoragePoolVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, masterVersion = 1), log id: 5cfee7ec >2014-07-25 21:22:28,120 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-15) Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Inactive to Unknown, reason : no reporting hosts >2014-07-25 21:23:21,976 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-12) FINISH, ConnectStoragePoolVDSCommand, log id: 5cfee7ec >2014-07-25 21:23:21,991 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-36) [b23fc58] START, SetMOMPolicyParametersVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 4a847345 >2014-07-25 21:23:22,085 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-36) [b23fc58] FINISH, SetMOMPolicyParametersVDSCommand, log id: 4a847345 >2014-07-25 21:23:22,096 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-36) [b23fc58] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host rack was set to Up. >2014-07-25 21:23:22,104 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-36) [b23fc58] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host rack. >2014-07-25 21:23:22,129 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-36) [2acc035c] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS >2014-07-25 21:23:28,245 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-28) [8ef1638] hostFromVds::selectedVds - rack, spmStatus Free, storage pool Default >2014-07-25 21:23:28,251 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-28) [8ef1638] starting spm on vds rack, storage pool Default, prevId -1, LVER -1 >2014-07-25 21:23:28,253 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-28) [8ef1638] START, SpmStartVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, prevId=-1, prevLVER=-1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 788cebfe >2014-07-25 21:23:28,301 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-28) [8ef1638] spmStart polling started: taskId = 5d4b0fec-6945-4715-9afd-83382a58244b >2014-07-25 21:23:29,307 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-28) [8ef1638] spmStart polling ended: taskId = 5d4b0fec-6945-4715-9afd-83382a58244b task status = finished >2014-07-25 21:23:29,364 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-28) [8ef1638] spmStart polling ended, spm status: SPM >2014-07-25 21:23:29,367 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-28) [8ef1638] START, HSMClearTaskVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, taskId=5d4b0fec-6945-4715-9afd-83382a58244b), log id: 17bf8e5c >2014-07-25 21:23:29,411 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-28) [8ef1638] FINISH, HSMClearTaskVDSCommand, log id: 17bf8e5c >2014-07-25 21:23:29,412 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-28) [8ef1638] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@6a0a0eba, log id: 788cebfe >2014-07-25 21:23:29,416 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-28) [8ef1638] Initialize Irs proxy from vds: 10.34.62.208 >2014-07-25 21:23:29,422 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-28) [8ef1638] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208). >2014-07-25 21:23:29,446 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-33) [8ef1638] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 501306e2 >2014-07-25 21:23:29,469 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-33) [8ef1638] -- executeIrsBrokerCommand: Attempting on storage pool 00000002-0002-0002-0002-0000000001d9 >2014-07-25 21:23:29,472 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-33) [8ef1638] START, HSMGetAllTasksInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 42f72653 >2014-07-25 21:23:29,478 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-33) [8ef1638] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 42f72653 >2014-07-25 21:23:29,479 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-33) [8ef1638] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 501306e2 >2014-07-25 21:23:29,480 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (org.ovirt.thread.pool-8-thread-33) [8ef1638] Discovered no tasks on Storage Pool Default >2014-07-25 21:23:54,340 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-3) Principal admin::internal synchronized >2014-07-25 21:23:59,251 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-40) Attempting to update VMs/Templates Ovf. >2014-07-25 22:23:54,345 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-36) [2acc035c] Principal admin::internal synchronized >2014-07-25 22:23:59,281 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-83) Attempting to update VMs/Templates Ovf. >2014-07-25 23:23:54,355 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-18) Principal admin::internal synchronized >2014-07-25 23:23:59,294 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-37) Attempting to update VMs/Templates Ovf. >2014-07-26 00:23:54,363 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-22) Principal admin::internal synchronized >2014-07-26 00:23:59,308 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-51) Attempting to update VMs/Templates Ovf. >2014-07-26 01:23:54,372 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-66) Principal admin::internal synchronized >2014-07-26 01:23:59,340 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-78) Attempting to update VMs/Templates Ovf. >2014-07-26 02:23:54,377 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-82) Principal admin::internal synchronized >2014-07-26 02:23:59,355 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-92) Attempting to update VMs/Templates Ovf. >2014-07-26 03:23:54,384 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-62) Principal admin::internal synchronized >2014-07-26 03:23:59,479 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-2) Attempting to update VMs/Templates Ovf. >2014-07-26 03:35:35,001 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-61) Start deleteAgedOutAuditLogs >2014-07-26 03:35:35,001 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-54) Start deleteAgedOutCommandEntities >2014-07-26 03:35:35,049 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-61) Finished deleteAgedOutAuditLogs >2014-07-26 03:35:35,051 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-54) Finished deleteAgedOutCommandEntities >2014-07-26 04:23:54,390 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-13) [4dca0641] Principal admin::internal synchronized >2014-07-26 04:23:59,494 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-99) Attempting to update VMs/Templates Ovf. >2014-07-26 05:23:54,397 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-11) Principal admin::internal synchronized >2014-07-26 05:23:59,508 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-79) Attempting to update VMs/Templates Ovf. >2014-07-26 06:23:54,408 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-9) Principal admin::internal synchronized >2014-07-26 06:23:59,524 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-56) Attempting to update VMs/Templates Ovf. >2014-07-26 06:28:41,941 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-30) domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 in problem. vds: rack >2014-07-26 06:30:13,582 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-66) Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2014-07-26 06:30:13,590 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-66) [2734597c] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-07-26 06:30:13,597 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-66) [2734597c] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Active to Unknown, reason : null >2014-07-26 06:30:13,618 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-66) [2734597c] Correlation ID: 2734597c, Call Stack: null, Custom Event ID: -1, Message: Invalid status on Data Center Default. Setting Data Center status to Non Responsive (On host rack, Error: Cannot access master storage domain). >2014-07-26 06:32:13,628 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-66) [2734597c] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2014-07-26 06:32:13,631 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-66) [2734597c] hostFromVds::selectedVds - rack, spmStatus returned null! >2014-07-26 06:32:13,632 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-66) [2734597c] IrsBroker::Failed::GetStoragePoolInfoVDS >2014-07-26 06:32:13,633 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-66) [2734597c] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2014-07-26 06:32:13,668 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-6) [5e9a1b3a] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage >2014-07-26 06:32:13,694 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-6) [5e9a1b3a] START, ResetIrsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, ignoreStopFailed = true), log id: 537acf4 >2014-07-26 06:32:13,700 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-6) [5e9a1b3a] START, SpmStopVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9), log id: 6063da94 >2014-07-26 06:32:13,706 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-6) [5e9a1b3a] SpmStopVDSCommand::Stopping SPM on vds rack, pool id 00000002-0002-0002-0002-0000000001d9 >2014-07-26 06:34:13,601 WARN [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-6) [5e9a1b3a] could not stop spm of pool 00000002-0002-0002-0002-0000000001d9 on vds 17f7fe56-5f32-40ef-b098-611b84484e0b - reason: org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcRunTimeException: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException >2014-07-26 06:34:13,603 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-6) [5e9a1b3a] FINISH, SpmStopVDSCommand, log id: 6063da94 >2014-07-26 06:34:13,602 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-18) Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: VDSNetworkException: org.apache.commons.httpclient.NoHttpResponseException: The server 10.34.62.208 failed to respond >2014-07-26 06:34:13,607 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-18) hostFromVds::selectedVds - rack, spmStatus returned null! >2014-07-26 06:34:13,608 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ListVDSCommand] (DefaultQuartzScheduler_Worker-89) Command ListVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-07-26 06:34:13,609 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-6) [5e9a1b3a] FINISH, ResetIrsVDSCommand, log id: 537acf4 >2014-07-26 06:34:13,611 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-89) Host rack is not responding. It will stay in Connecting state for a grace period of $80 seconds and after that an attempt to fence the host will be issued. >2014-07-26 06:34:13,613 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-89) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:406) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.fetchRunningVms(VdsUpdateRunTimeInfo.java:971) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVmStats(VdsUpdateRunTimeInfo.java:920) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:638) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:235) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) [:1.7.0_55] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_55] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_55] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_55] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_55] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_55] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_55] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_55] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy82.list(Unknown Source) > at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source) [:1.7.0_55] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_55] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_55] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_55] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_55] > >2014-07-26 06:34:13,634 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-6) [5e9a1b3a] Correlation ID: 5e9a1b3a, Job ID: b98e617d-8439-4ba7-bf5d-74d935be6985, Call Stack: null, Custom Event ID: -1, Message: Failed to Reconstruct Master Domain for Data Center Default. >2014-07-26 06:34:13,654 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-10) [2734597c] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue >2014-07-26 06:34:13,657 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-47) starting processDomainRecovery for domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 >2014-07-26 06:34:13,660 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-89) Failed to refresh VDS , vds = 17f7fe56-5f32-40ef-b098-611b84484e0b : rack, VDS Network Error, continuing. >java.net.ConnectException: Connection refused >2014-07-26 06:34:13,731 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-47) Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 was reported by all hosts in status UP as problematic. Not moving the domain to NonOperational because it is being reconstructed now. >2014-07-26 06:34:13,749 WARN [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-47) [7e45f3f9] CanDoAction of action ReconstructMasterDomain failed. Reasons:VAR__ACTION__RECONSTRUCT_MASTER,VAR__TYPE__STORAGE__DOMAIN,ACTION_TYPE_FAILED_NO_VDS_IN_POOL >2014-07-26 06:34:13,757 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-47) [7e45f3f9] Removing vds [] from the domain in maintenance cache >2014-07-26 06:34:13,758 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-47) [7e45f3f9] Removing host(s) [] from hosts unseen domain report cache >2014-07-26 06:34:13,758 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-10) [2734597c] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue >2014-07-26 06:34:16,766 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-16) Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@e6ebdf4 >2014-07-26 06:34:16,769 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-16) HostName = rack >2014-07-26 06:34:16,771 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-16) Command GetCapabilitiesVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]) execution failed. Exception: VDSRecoveringException: Recovering from crash or Initializing >2014-07-26 06:34:16,790 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-16) Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSRecoveringException: Recovering from crash or Initializing > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:16) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:566) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:235) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Host rack is initializing. Message: Recovering from crash or Initializing >2014-07-26 06:34:16,799 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-16) Failed to refresh VDS , vds = 17f7fe56-5f32-40ef-b098-611b84484e0b : rack, error = Recovering from crash or Initializing, continuing. >2014-07-26 06:34:20,035 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-58) [266916e1] START, GetHardwareInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]), log id: 2efd43b >2014-07-26 06:34:20,086 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-58) [266916e1] FINISH, GetHardwareInfoVDSCommand, log id: 2efd43b >2014-07-26 06:34:20,128 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-58) [43049438] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS >2014-07-26 06:34:20,142 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-58) [43049438] Cancelling the recovery from crash timer for VDS 17f7fe56-5f32-40ef-b098-611b84484e0b because vds started initializing >2014-07-26 06:34:20,145 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-58) [222469a6] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-07-26 06:34:20,154 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-58) [75046dac] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-07-26 06:34:20,165 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-58) [75046dac] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, storageType = NFS, connectionList = [{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 2025934c >2014-07-26 06:34:20,218 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-58) [75046dac] FINISH, ConnectStorageServerVDSCommand, return: {5083f370-af09-457b-9b73-3440e8b65a07=0}, log id: 2025934c >2014-07-26 06:34:20,220 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-58) [75046dac] Host rack storage connection was succeeded >2014-07-26 06:34:20,227 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-42) START, ConnectStoragePoolVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, masterVersion = 1), log id: 65f4633d >2014-07-26 06:34:23,616 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-22) Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Inactive to Unknown, reason : no reporting hosts >2014-07-26 06:35:20,841 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-42) FINISH, ConnectStoragePoolVDSCommand, log id: 65f4633d >2014-07-26 06:35:20,854 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-58) [75046dac] START, SetMOMPolicyParametersVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 381d3566 >2014-07-26 06:35:20,947 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-58) [75046dac] FINISH, SetMOMPolicyParametersVDSCommand, log id: 381d3566 >2014-07-26 06:35:20,954 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-58) [75046dac] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host rack was set to Up. >2014-07-26 06:35:20,960 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-58) [75046dac] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host rack. >2014-07-26 06:35:20,975 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-58) [3d5a7fcf] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS >2014-07-26 06:35:24,555 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-92) hostFromVds::selectedVds - rack, spmStatus Free, storage pool Default >2014-07-26 06:35:24,561 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-92) starting spm on vds rack, storage pool Default, prevId -1, LVER -1 >2014-07-26 06:35:24,563 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-92) START, SpmStartVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, prevId=-1, prevLVER=-1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 6952f735 >2014-07-26 06:35:24,612 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-92) spmStart polling started: taskId = adefb6d5-a768-45d8-9dce-959bff0b1479 >2014-07-26 06:35:26,624 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-92) spmStart polling ended: taskId = adefb6d5-a768-45d8-9dce-959bff0b1479 task status = finished >2014-07-26 06:35:26,679 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-92) spmStart polling ended, spm status: SPM >2014-07-26 06:35:26,682 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-92) START, HSMClearTaskVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, taskId=adefb6d5-a768-45d8-9dce-959bff0b1479), log id: 40a12104 >2014-07-26 06:35:26,727 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-92) FINISH, HSMClearTaskVDSCommand, log id: 40a12104 >2014-07-26 06:35:26,728 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-92) FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@574079b, log id: 6952f735 >2014-07-26 06:35:26,732 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-92) Initialize Irs proxy from vds: 10.34.62.208 >2014-07-26 06:35:26,738 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-92) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208). >2014-07-26 06:35:26,740 WARN [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-8-thread-2) Executing a command: java.util.concurrent.FutureTask , but note that there are 1 tasks in the queue. >2014-07-26 06:35:26,750 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-2) START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 5c47d77f >2014-07-26 06:35:26,782 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-2) -- executeIrsBrokerCommand: Attempting on storage pool 00000002-0002-0002-0002-0000000001d9 >2014-07-26 06:35:26,783 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-2) START, HSMGetAllTasksInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 69e92642 >2014-07-26 06:35:26,788 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-2) FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 69e92642 >2014-07-26 06:35:26,789 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-2) FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 5c47d77f >2014-07-26 06:35:26,790 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (org.ovirt.thread.pool-8-thread-2) Discovered no tasks on Storage Pool Default >2014-07-26 07:23:54,414 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-24) Principal admin::internal synchronized >2014-07-26 07:23:59,559 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-33) Attempting to update VMs/Templates Ovf. >2014-07-26 08:23:54,419 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-85) Principal admin::internal synchronized >2014-07-26 08:23:59,575 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-81) Attempting to update VMs/Templates Ovf. >2014-07-26 09:01:58,120 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-67) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage domain 35 experienced a high latency of 5.61273 seconds from host rack. This may cause performance and functional issues. Please consult your Storage Administrator. >2014-07-26 09:23:54,427 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-80) Principal admin::internal synchronized >2014-07-26 09:23:59,591 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-25) Attempting to update VMs/Templates Ovf. >2014-07-26 10:23:54,434 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-67) Principal admin::internal synchronized >2014-07-26 10:23:59,608 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-98) Attempting to update VMs/Templates Ovf. >2014-07-26 11:23:54,442 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-3) Principal admin::internal synchronized >2014-07-26 11:23:59,636 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-34) Attempting to update VMs/Templates Ovf. >2014-07-26 12:23:54,458 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-25) Principal admin::internal synchronized >2014-07-26 12:23:59,654 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-15) Attempting to update VMs/Templates Ovf. >2014-07-26 13:23:54,466 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-21) Principal admin::internal synchronized >2014-07-26 13:23:59,673 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-56) Attempting to update VMs/Templates Ovf. >2014-07-26 14:23:54,475 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-84) Principal admin::internal synchronized >2014-07-26 14:23:59,705 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-38) [1b9ef0f] Attempting to update VMs/Templates Ovf. >2014-07-26 15:23:54,481 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-62) Principal admin::internal synchronized >2014-07-26 15:23:59,720 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-86) Attempting to update VMs/Templates Ovf. >2014-07-26 15:56:16,647 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-11) domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 in problem. vds: rack >2014-07-26 15:57:54,449 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-82) Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2014-07-26 15:57:54,454 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-82) [3abdaaa6] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-07-26 15:57:54,458 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-82) [3abdaaa6] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Active to Unknown, reason : null >2014-07-26 15:57:54,472 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-82) [3abdaaa6] Correlation ID: 3abdaaa6, Call Stack: null, Custom Event ID: -1, Message: Invalid status on Data Center Default. Setting Data Center status to Non Responsive (On host rack, Error: Cannot access master storage domain). >2014-07-26 15:59:54,532 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-82) [3abdaaa6] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2014-07-26 15:59:54,535 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-82) [3abdaaa6] hostFromVds::selectedVds - rack, spmStatus returned null! >2014-07-26 15:59:54,536 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-82) [3abdaaa6] IrsBroker::Failed::GetStoragePoolInfoVDS >2014-07-26 15:59:54,537 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-82) [3abdaaa6] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2014-07-26 15:59:54,582 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-1) [6d2260aa] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage >2014-07-26 15:59:54,601 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-1) [6d2260aa] START, ResetIrsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, ignoreStopFailed = true), log id: 232547d3 >2014-07-26 15:59:54,605 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-1) [6d2260aa] START, SpmStopVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9), log id: 29c94738 >2014-07-26 15:59:54,611 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-1) [6d2260aa] SpmStopVDSCommand::Stopping SPM on vds rack, pool id 00000002-0002-0002-0002-0000000001d9 >2014-07-26 16:00:00,005 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-61) Autorecovering 1 storage domains >2014-07-26 16:00:00,006 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-61) Autorecovering storage domains id: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 >2014-07-26 16:00:00,009 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-61) [58bfe349] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage >2014-07-26 16:00:00,011 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-61) [58bfe349] ConnectDomainToStorage. Before Connect all hosts to pool. Time:7/26/14 4:00 PM >2014-07-26 16:00:00,019 INFO [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-8-thread-11) [1c2fde6c] Running command: ConnectStorageToVdsCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2014-07-26 16:00:00,023 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-11) [1c2fde6c] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 59fe87bd >2014-07-26 16:01:00,035 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-11) [1c2fde6c] FINISH, ConnectStorageServerVDSCommand, return: {5083f370-af09-457b-9b73-3440e8b65a07=0}, log id: 59fe87bd >2014-07-26 16:01:00,039 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-61) [58bfe349] ConnectDomainToStorage. After Connect all hosts to pool. Time:7/26/14 4:01 PM >2014-07-26 16:01:54,501 WARN [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-1) [6d2260aa] could not stop spm of pool 00000002-0002-0002-0002-0000000001d9 on vds 17f7fe56-5f32-40ef-b098-611b84484e0b - reason: org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcRunTimeException: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException >2014-07-26 16:01:54,504 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-25) Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: VDSNetworkException: org.apache.commons.httpclient.NoHttpResponseException: The server 10.34.62.208 failed to respond >2014-07-26 16:01:54,506 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-25) hostFromVds::selectedVds - rack, spmStatus returned null! >2014-07-26 16:01:54,503 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetStatsVDSCommand] (DefaultQuartzScheduler_Worker-21) Command GetStatsVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-07-26 16:01:54,504 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-1) [6d2260aa] FINISH, SpmStopVDSCommand, log id: 29c94738 >2014-07-26 16:01:54,512 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-21) Host rack is not responding. It will stay in Connecting state for a grace period of $80 seconds and after that an attempt to fence the host will be issued. >2014-07-26 16:01:54,513 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-21) vds::refreshVdsStats Failed getVdsStats, vds = 17f7fe56-5f32-40ef-b098-611b84484e0b : rack, error = VDSNetworkException: java.net.ConnectException: Connection refused >2014-07-26 16:01:54,515 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-21) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsStats(VdsUpdateRunTimeInfo.java:669) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:623) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:235) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) [:1.7.0_55] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_55] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_55] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_55] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_55] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_55] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_55] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_55] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy82.getVdsStats(Unknown Source) > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_55] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_55] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_55] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_55] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_55] > >2014-07-26 16:01:54,519 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-1) [6d2260aa] FINISH, ResetIrsVDSCommand, log id: 232547d3 >2014-07-26 16:01:54,545 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-21) Failed to refresh VDS , vds = 17f7fe56-5f32-40ef-b098-611b84484e0b : rack, VDS Network Error, continuing. >java.net.ConnectException: Connection refused >2014-07-26 16:01:54,559 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-1) [6d2260aa] Correlation ID: 6d2260aa, Job ID: 30d80a85-17b3-4966-9eab-0b2c0c124cc7, Call Stack: null, Custom Event ID: -1, Message: Failed to Reconstruct Master Domain for Data Center Default. >2014-07-26 16:01:54,568 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-41) [3abdaaa6] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue >2014-07-26 16:01:54,571 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-7) starting processDomainRecovery for domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 >2014-07-26 16:01:54,641 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-7) Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 was reported by all hosts in status UP as problematic. Not moving the domain to NonOperational because it is being reconstructed now. >2014-07-26 16:01:54,667 WARN [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-7) [8e871e1] CanDoAction of action ReconstructMasterDomain failed. Reasons:VAR__ACTION__RECONSTRUCT_MASTER,VAR__TYPE__STORAGE__DOMAIN,ACTION_TYPE_FAILED_NO_VDS_IN_POOL >2014-07-26 16:01:54,674 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-7) [8e871e1] Removing vds [] from the domain in maintenance cache >2014-07-26 16:01:54,675 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-7) [8e871e1] Removing host(s) [] from hosts unseen domain report cache >2014-07-26 16:01:54,676 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-41) [3abdaaa6] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue >2014-07-26 16:01:57,629 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-92) Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@5309f983 >2014-07-26 16:01:57,631 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-92) HostName = rack >2014-07-26 16:01:57,631 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-92) Command GetCapabilitiesVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]) execution failed. Exception: VDSRecoveringException: Recovering from crash or Initializing >2014-07-26 16:01:57,649 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-92) Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSRecoveringException: Recovering from crash or Initializing > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:16) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:566) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:235) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Host rack is initializing. Message: Recovering from crash or Initializing >2014-07-26 16:01:57,657 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-92) Failed to refresh VDS , vds = 17f7fe56-5f32-40ef-b098-611b84484e0b : rack, error = Recovering from crash or Initializing, continuing. >2014-07-26 16:02:00,904 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-36) [2acc035c] START, GetHardwareInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]), log id: 644182f9 >2014-07-26 16:02:00,955 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-36) [2acc035c] FINISH, GetHardwareInfoVDSCommand, log id: 644182f9 >2014-07-26 16:02:00,974 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-36) [51c47cce] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS >2014-07-26 16:02:01,033 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-36) [51c47cce] Cancelling the recovery from crash timer for VDS 17f7fe56-5f32-40ef-b098-611b84484e0b because vds started initializing >2014-07-26 16:02:01,035 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-36) [2d25bc14] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-07-26 16:02:01,042 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-36) [4d34b8ed] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-07-26 16:02:01,050 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-36) [4d34b8ed] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, storageType = NFS, connectionList = [{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 67765276 >2014-07-26 16:02:01,103 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-36) [4d34b8ed] FINISH, ConnectStorageServerVDSCommand, return: {5083f370-af09-457b-9b73-3440e8b65a07=0}, log id: 67765276 >2014-07-26 16:02:01,104 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-36) [4d34b8ed] Host rack storage connection was succeeded >2014-07-26 16:02:01,109 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-40) START, ConnectStoragePoolVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, masterVersion = 1), log id: 3ed40812 >2014-07-26 16:02:04,511 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-96) Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Inactive to Unknown, reason : no reporting hosts >2014-07-26 16:03:08,236 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-40) FINISH, ConnectStoragePoolVDSCommand, log id: 3ed40812 >2014-07-26 16:03:08,249 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-36) [4d34b8ed] START, SetMOMPolicyParametersVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 1351da37 >2014-07-26 16:03:08,343 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-36) [4d34b8ed] FINISH, SetMOMPolicyParametersVDSCommand, log id: 1351da37 >2014-07-26 16:03:08,351 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-36) [4d34b8ed] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host rack was set to Up. >2014-07-26 16:03:08,357 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-36) [4d34b8ed] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host rack. >2014-07-26 16:03:08,378 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-36) [3e2ffa8] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS >2014-07-26 16:03:14,855 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-68) hostFromVds::selectedVds - rack, spmStatus Free, storage pool Default >2014-07-26 16:03:14,861 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-68) starting spm on vds rack, storage pool Default, prevId -1, LVER -1 >2014-07-26 16:03:14,864 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-68) START, SpmStartVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, prevId=-1, prevLVER=-1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 39164ee7 >2014-07-26 16:03:14,914 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-68) spmStart polling started: taskId = 9cbbbfc5-51f2-47e9-890c-ea207dca555d >2014-07-26 16:03:17,931 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-68) spmStart polling ended: taskId = 9cbbbfc5-51f2-47e9-890c-ea207dca555d task status = finished >2014-07-26 16:03:18,017 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-68) spmStart polling ended, spm status: SPM >2014-07-26 16:03:18,020 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-68) START, HSMClearTaskVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, taskId=9cbbbfc5-51f2-47e9-890c-ea207dca555d), log id: 7a736be4 >2014-07-26 16:03:18,065 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-68) FINISH, HSMClearTaskVDSCommand, log id: 7a736be4 >2014-07-26 16:03:18,067 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-68) FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@289da6d6, log id: 39164ee7 >2014-07-26 16:03:18,071 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-68) Initialize Irs proxy from vds: 10.34.62.208 >2014-07-26 16:03:18,078 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-68) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208). >2014-07-26 16:03:18,080 WARN [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-8-thread-46) Executing a command: java.util.concurrent.FutureTask , but note that there are 1 tasks in the queue. >2014-07-26 16:03:18,095 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-46) START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 2580f4f8 >2014-07-26 16:03:18,161 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-46) -- executeIrsBrokerCommand: Attempting on storage pool 00000002-0002-0002-0002-0000000001d9 >2014-07-26 16:03:18,164 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-46) START, HSMGetAllTasksInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 275457d3 >2014-07-26 16:03:18,169 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-46) FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 275457d3 >2014-07-26 16:03:18,170 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-46) FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 2580f4f8 >2014-07-26 16:03:18,171 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (org.ovirt.thread.pool-8-thread-46) Discovered no tasks on Storage Pool Default >2014-07-26 16:23:54,492 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-18) Principal admin::internal synchronized >2014-07-26 16:23:59,752 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-22) Attempting to update VMs/Templates Ovf. >2014-07-26 17:23:54,502 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-21) Principal admin::internal synchronized >2014-07-26 17:23:59,784 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-9) Attempting to update VMs/Templates Ovf. >2014-07-26 18:23:54,507 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-95) Principal admin::internal synchronized >2014-07-26 18:23:59,801 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-69) Attempting to update VMs/Templates Ovf. >2014-07-26 19:23:54,516 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-5) Principal admin::internal synchronized >2014-07-26 19:23:59,818 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-25) Attempting to update VMs/Templates Ovf. >2014-07-26 20:23:54,522 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-85) Principal admin::internal synchronized >2014-07-26 20:23:59,833 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-96) Attempting to update VMs/Templates Ovf. >2014-07-26 21:23:54,528 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-42) Principal admin::internal synchronized >2014-07-26 21:23:59,864 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-49) Attempting to update VMs/Templates Ovf. >2014-07-26 22:23:54,535 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-33) Principal admin::internal synchronized >2014-07-26 22:23:59,876 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-21) Attempting to update VMs/Templates Ovf. >2014-07-26 23:23:54,544 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-31) Principal admin::internal synchronized >2014-07-26 23:23:59,893 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-43) Attempting to update VMs/Templates Ovf. >2014-07-27 00:23:54,549 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-22) Principal admin::internal synchronized >2014-07-27 00:23:59,906 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-98) Attempting to update VMs/Templates Ovf. >2014-07-27 01:19:29,341 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-46) domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 in problem. vds: rack >2014-07-27 01:19:59,582 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-12) Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 recovered from problem. vds: rack >2014-07-27 01:19:59,583 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-12) Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 has recovered from problem. No active host in the DC is reporting it as problematic, so clearing the domain recovery timer. >2014-07-27 01:23:54,555 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-1) Principal admin::internal synchronized >2014-07-27 01:23:59,922 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-12) Attempting to update VMs/Templates Ovf. >2014-07-27 01:24:31,796 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-15) domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 in problem. vds: rack >2014-07-27 01:26:05,680 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-93) Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2014-07-27 01:26:05,685 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-93) [341aec49] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-07-27 01:26:05,689 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-93) [341aec49] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Active to Unknown, reason : null >2014-07-27 01:26:05,706 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-93) [341aec49] Correlation ID: 341aec49, Call Stack: null, Custom Event ID: -1, Message: Invalid status on Data Center Default. Setting Data Center status to Non Responsive (On host rack, Error: Cannot access master storage domain). >2014-07-27 01:28:05,729 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-93) [341aec49] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2014-07-27 01:28:05,731 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-93) [341aec49] hostFromVds::selectedVds - rack, spmStatus returned null! >2014-07-27 01:28:05,733 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-93) [341aec49] IrsBroker::Failed::GetStoragePoolInfoVDS >2014-07-27 01:28:05,737 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-93) [341aec49] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2014-07-27 01:28:05,808 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-4) [7fa5cf57] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage >2014-07-27 01:28:05,929 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-4) [7fa5cf57] START, ResetIrsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, ignoreStopFailed = true), log id: 1e8bf405 >2014-07-27 01:28:05,932 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-4) [7fa5cf57] START, SpmStopVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9), log id: 49f1e432 >2014-07-27 01:28:05,938 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-4) [7fa5cf57] SpmStopVDSCommand::Stopping SPM on vds rack, pool id 00000002-0002-0002-0002-0000000001d9 >2014-07-27 01:30:00,006 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-50) Autorecovering 1 storage domains >2014-07-27 01:30:00,008 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-50) Autorecovering storage domains id: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 >2014-07-27 01:30:00,011 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-50) [375fcc6] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage >2014-07-27 01:30:00,015 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-50) [375fcc6] ConnectDomainToStorage. Before Connect all hosts to pool. Time:7/27/14 1:30 AM >2014-07-27 01:30:00,026 INFO [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-8-thread-34) [390d2a31] Running command: ConnectStorageToVdsCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2014-07-27 01:30:00,030 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-34) [390d2a31] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 179e9b21 >2014-07-27 01:30:05,699 WARN [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-4) [7fa5cf57] could not stop spm of pool 00000002-0002-0002-0002-0000000001d9 on vds 17f7fe56-5f32-40ef-b098-611b84484e0b - reason: org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcRunTimeException: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException >2014-07-27 01:30:05,701 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-4) [7fa5cf57] FINISH, SpmStopVDSCommand, log id: 49f1e432 >2014-07-27 01:30:05,706 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-35) Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: VDSNetworkException: org.apache.commons.httpclient.NoHttpResponseException: The server 10.34.62.208 failed to respond >2014-07-27 01:30:05,708 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-35) hostFromVds::selectedVds - rack, spmStatus returned null! >2014-07-27 01:30:05,711 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetStatsVDSCommand] (DefaultQuartzScheduler_Worker-2) Command GetStatsVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-07-27 01:30:05,715 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-34) [390d2a31] Command ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-07-27 01:30:05,718 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-34) [390d2a31] FINISH, ConnectStorageServerVDSCommand, log id: 179e9b21 >2014-07-27 01:30:05,716 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-2) Host rack is not responding. It will stay in Connecting state for a grace period of $80 seconds and after that an attempt to fence the host will be issued. >2014-07-27 01:30:05,720 ERROR [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-8-thread-34) [390d2a31] Command org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-07-27 01:30:05,722 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-4) [7fa5cf57] FINISH, ResetIrsVDSCommand, log id: 1e8bf405 >2014-07-27 01:30:05,721 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-2) vds::refreshVdsStats Failed getVdsStats, vds = 17f7fe56-5f32-40ef-b098-611b84484e0b : rack, error = VDSNetworkException: java.net.ConnectException: Connection refused >2014-07-27 01:30:05,726 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-2) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsStats(VdsUpdateRunTimeInfo.java:669) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:623) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:235) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) [:1.7.0_55] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_55] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_55] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_55] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_55] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_55] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_55] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_55] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy82.getVdsStats(Unknown Source) > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_55] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_55] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_55] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_55] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_55] > >2014-07-27 01:30:05,741 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-4) [7fa5cf57] Correlation ID: 7fa5cf57, Job ID: 2dc1eb12-7c29-446e-95d6-40b9981246dc, Call Stack: null, Custom Event ID: -1, Message: Failed to Reconstruct Master Domain for Data Center Default. >2014-07-27 01:30:05,734 ERROR [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-8-thread-34) [390d2a31] Transaction rolled-back for command: org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand. >2014-07-27 01:30:05,758 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-50) [375fcc6] ConnectDomainToStorage. After Connect all hosts to pool. Time:7/27/14 1:30 AM >2014-07-27 01:30:05,763 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-2) Failed to refresh VDS , vds = 17f7fe56-5f32-40ef-b098-611b84484e0b : rack, VDS Network Error, continuing. >java.net.ConnectException: Connection refused >2014-07-27 01:30:05,764 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-29) [341aec49] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue >2014-07-27 01:30:05,766 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-21) starting processDomainRecovery for domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 >2014-07-27 01:30:05,826 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-21) Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 was reported by all hosts in status UP as problematic. Not moving the domain to NonOperational because it is being reconstructed now. >2014-07-27 01:30:05,849 WARN [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-21) [68bbec00] CanDoAction of action ReconstructMasterDomain failed. Reasons:VAR__ACTION__RECONSTRUCT_MASTER,VAR__TYPE__STORAGE__DOMAIN,ACTION_TYPE_FAILED_NO_VDS_IN_POOL >2014-07-27 01:30:05,857 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-21) [68bbec00] Removing vds [] from the domain in maintenance cache >2014-07-27 01:30:05,857 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-21) [68bbec00] Removing host(s) [] from hosts unseen domain report cache >2014-07-27 01:30:05,858 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-29) [341aec49] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue >2014-07-27 01:30:08,850 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-56) Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@b06f5c3 >2014-07-27 01:30:08,852 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-56) HostName = rack >2014-07-27 01:30:08,853 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-56) Command GetCapabilitiesVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]) execution failed. Exception: VDSRecoveringException: Recovering from crash or Initializing >2014-07-27 01:30:08,869 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-56) Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSRecoveringException: Recovering from crash or Initializing > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:16) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:566) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:235) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Host rack is initializing. Message: Recovering from crash or Initializing >2014-07-27 01:30:08,876 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-56) Failed to refresh VDS , vds = 17f7fe56-5f32-40ef-b098-611b84484e0b : rack, error = Recovering from crash or Initializing, continuing. >2014-07-27 01:30:12,110 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-44) START, GetHardwareInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]), log id: 5adc3365 >2014-07-27 01:30:12,158 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-44) FINISH, GetHardwareInfoVDSCommand, log id: 5adc3365 >2014-07-27 01:30:12,195 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-44) [3af0095b] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS >2014-07-27 01:30:12,204 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-44) [3af0095b] Cancelling the recovery from crash timer for VDS 17f7fe56-5f32-40ef-b098-611b84484e0b because vds started initializing >2014-07-27 01:30:12,206 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-44) [16ac6dbe] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-07-27 01:30:12,214 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-44) [22e0a75a] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-07-27 01:30:12,226 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-44) [22e0a75a] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, storageType = NFS, connectionList = [{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 754cd96 >2014-07-27 01:30:12,279 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-44) [22e0a75a] FINISH, ConnectStorageServerVDSCommand, return: {5083f370-af09-457b-9b73-3440e8b65a07=0}, log id: 754cd96 >2014-07-27 01:30:12,280 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-44) [22e0a75a] Host rack storage connection was succeeded >2014-07-27 01:30:12,288 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-48) START, ConnectStoragePoolVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, masterVersion = 1), log id: 27b06332 >2014-07-27 01:30:15,714 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-19) Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Inactive to Unknown, reason : no reporting hosts >2014-07-27 01:31:12,708 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-48) FINISH, ConnectStoragePoolVDSCommand, log id: 27b06332 >2014-07-27 01:31:12,721 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-44) [22e0a75a] START, SetMOMPolicyParametersVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 34c8d082 >2014-07-27 01:31:12,814 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-44) [22e0a75a] FINISH, SetMOMPolicyParametersVDSCommand, log id: 34c8d082 >2014-07-27 01:31:12,822 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-44) [22e0a75a] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host rack was set to Up. >2014-07-27 01:31:12,828 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-44) [22e0a75a] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host rack. >2014-07-27 01:31:12,843 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-44) [60dd31b0] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS >2014-07-27 01:31:15,829 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-31) hostFromVds::selectedVds - rack, spmStatus Free, storage pool Default >2014-07-27 01:31:15,835 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-31) starting spm on vds rack, storage pool Default, prevId -1, LVER -1 >2014-07-27 01:31:15,837 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-31) START, SpmStartVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, prevId=-1, prevLVER=-1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 4fdd429a >2014-07-27 01:31:15,885 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-31) spmStart polling started: taskId = c940b182-f451-44f2-8e44-2741b374ae40 >2014-07-27 01:31:16,894 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-31) spmStart polling ended: taskId = c940b182-f451-44f2-8e44-2741b374ae40 task status = finished >2014-07-27 01:31:16,950 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-31) spmStart polling ended, spm status: SPM >2014-07-27 01:31:16,953 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-31) START, HSMClearTaskVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, taskId=c940b182-f451-44f2-8e44-2741b374ae40), log id: 445c5bd6 >2014-07-27 01:31:16,997 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-31) FINISH, HSMClearTaskVDSCommand, log id: 445c5bd6 >2014-07-27 01:31:16,999 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-31) FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@151da3d2, log id: 4fdd429a >2014-07-27 01:31:17,003 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-31) Initialize Irs proxy from vds: 10.34.62.208 >2014-07-27 01:31:17,009 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-31) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208). >2014-07-27 01:31:17,020 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-20) START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 10d611 >2014-07-27 01:31:17,043 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-20) -- executeIrsBrokerCommand: Attempting on storage pool 00000002-0002-0002-0002-0000000001d9 >2014-07-27 01:31:17,045 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-20) START, HSMGetAllTasksInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 2f95a703 >2014-07-27 01:31:17,051 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-20) FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 2f95a703 >2014-07-27 01:31:17,052 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-20) FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 10d611 >2014-07-27 01:31:17,053 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (org.ovirt.thread.pool-8-thread-20) Discovered no tasks on Storage Pool Default >2014-07-27 02:23:54,562 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-46) Principal admin::internal synchronized >2014-07-27 02:23:59,938 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-5) Attempting to update VMs/Templates Ovf. >2014-07-27 03:23:54,572 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-65) Principal admin::internal synchronized >2014-07-27 03:23:59,971 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-57) Attempting to update VMs/Templates Ovf. >2014-07-27 03:35:35,000 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-86) Start deleteAgedOutAuditLogs >2014-07-27 03:35:35,001 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-15) Start deleteAgedOutCommandEntities >2014-07-27 03:35:35,006 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-86) Finished deleteAgedOutAuditLogs >2014-07-27 03:35:35,008 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-15) Finished deleteAgedOutCommandEntities >2014-07-27 04:23:54,579 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-43) Principal admin::internal synchronized >2014-07-27 04:24:00,003 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-60) Attempting to update VMs/Templates Ovf. >2014-07-27 05:23:54,589 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-22) Principal admin::internal synchronized >2014-07-27 05:24:00,019 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-48) Attempting to update VMs/Templates Ovf. >2014-07-27 06:00:30,041 WARN [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-8-thread-42) Executing a command: java.util.concurrent.FutureTask , but note that there are 1 tasks in the queue. >2014-07-27 06:23:54,595 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-76) Principal admin::internal synchronized >2014-07-27 06:24:00,050 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-84) Attempting to update VMs/Templates Ovf. >2014-07-27 07:23:54,601 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-29) Principal admin::internal synchronized >2014-07-27 07:24:00,066 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-34) Attempting to update VMs/Templates Ovf. >2014-07-27 08:23:54,610 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-50) [375fcc6] Principal admin::internal synchronized >2014-07-27 08:24:00,080 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-51) Attempting to update VMs/Templates Ovf. >2014-07-27 09:23:54,617 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-20) Principal admin::internal synchronized >2014-07-27 09:24:00,110 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-16) Attempting to update VMs/Templates Ovf. >2014-07-27 10:23:54,623 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-46) Principal admin::internal synchronized >2014-07-27 10:24:00,123 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-27) Attempting to update VMs/Templates Ovf. >2014-07-27 10:37:28,171 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-48) domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 in problem. vds: rack >2014-07-27 10:39:03,258 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-62) Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2014-07-27 10:39:03,270 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-62) [2e742a17] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-07-27 10:39:03,276 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-62) [2e742a17] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Active to Unknown, reason : null >2014-07-27 10:39:03,292 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-62) [2e742a17] Correlation ID: 2e742a17, Call Stack: null, Custom Event ID: -1, Message: Invalid status on Data Center Default. Setting Data Center status to Non Responsive (On host rack, Error: Cannot access master storage domain). >2014-07-27 10:41:03,307 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-62) [2e742a17] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2014-07-27 10:41:03,310 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-62) [2e742a17] hostFromVds::selectedVds - rack, spmStatus returned null! >2014-07-27 10:41:03,311 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-62) [2e742a17] IrsBroker::Failed::GetStoragePoolInfoVDS >2014-07-27 10:41:03,312 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-62) [2e742a17] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2014-07-27 10:41:03,358 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-28) [606e9025] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage >2014-07-27 10:41:03,375 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-28) [606e9025] START, ResetIrsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, ignoreStopFailed = true), log id: 56499c77 >2014-07-27 10:41:03,379 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-28) [606e9025] START, SpmStopVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9), log id: 591fa6b6 >2014-07-27 10:41:03,385 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-28) [606e9025] SpmStopVDSCommand::Stopping SPM on vds rack, pool id 00000002-0002-0002-0002-0000000001d9 >2014-07-27 10:43:03,281 WARN [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-28) [606e9025] could not stop spm of pool 00000002-0002-0002-0002-0000000001d9 on vds 17f7fe56-5f32-40ef-b098-611b84484e0b - reason: org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcRunTimeException: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException >2014-07-27 10:43:03,284 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-28) [606e9025] FINISH, SpmStopVDSCommand, log id: 591fa6b6 >2014-07-27 10:43:03,287 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetStatsVDSCommand] (DefaultQuartzScheduler_Worker-47) Command GetStatsVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-07-27 10:43:03,290 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-47) Host rack is not responding. It will stay in Connecting state for a grace period of $80 seconds and after that an attempt to fence the host will be issued. >2014-07-27 10:43:03,292 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-82) [3abdaaa6] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: VDSNetworkException: org.apache.commons.httpclient.NoHttpResponseException: The server 10.34.62.208 failed to respond >2014-07-27 10:43:03,295 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-82) [3abdaaa6] hostFromVds::selectedVds - rack, spmStatus returned null! >2014-07-27 10:43:03,294 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-47) vds::refreshVdsStats Failed getVdsStats, vds = 17f7fe56-5f32-40ef-b098-611b84484e0b : rack, error = VDSNetworkException: java.net.ConnectException: Connection refused >2014-07-27 10:43:03,294 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-28) [606e9025] FINISH, ResetIrsVDSCommand, log id: 56499c77 >2014-07-27 10:43:03,309 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-28) [606e9025] Correlation ID: 606e9025, Job ID: 4a0911e4-efb5-4e19-a046-77c333224b30, Call Stack: null, Custom Event ID: -1, Message: Failed to Reconstruct Master Domain for Data Center Default. >2014-07-27 10:43:03,297 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-47) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsStats(VdsUpdateRunTimeInfo.java:669) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:623) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:235) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) [:1.7.0_55] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_55] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_55] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_55] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_55] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_55] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_55] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_55] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy82.getVdsStats(Unknown Source) > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_55] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_55] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_55] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_55] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_55] > >2014-07-27 10:43:03,319 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-49) [2e742a17] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue >2014-07-27 10:43:03,341 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-1) starting processDomainRecovery for domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 >2014-07-27 10:43:03,344 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-47) Failed to refresh VDS , vds = 17f7fe56-5f32-40ef-b098-611b84484e0b : rack, VDS Network Error, continuing. >java.net.ConnectException: Connection refused >2014-07-27 10:43:03,402 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-1) Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 was reported by all hosts in status UP as problematic. Not moving the domain to NonOperational because it is being reconstructed now. >2014-07-27 10:43:03,429 WARN [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-1) [f01a1d3] CanDoAction of action ReconstructMasterDomain failed. Reasons:VAR__ACTION__RECONSTRUCT_MASTER,VAR__TYPE__STORAGE__DOMAIN,ACTION_TYPE_FAILED_NO_VDS_IN_POOL >2014-07-27 10:43:03,436 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-1) [f01a1d3] Removing vds [] from the domain in maintenance cache >2014-07-27 10:43:03,437 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-1) [f01a1d3] Removing host(s) [] from hosts unseen domain report cache >2014-07-27 10:43:03,438 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-49) [2e742a17] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue >2014-07-27 10:43:06,847 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-98) START, GetHardwareInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]), log id: 61f0f4eb >2014-07-27 10:43:06,899 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-98) FINISH, GetHardwareInfoVDSCommand, log id: 61f0f4eb >2014-07-27 10:43:06,925 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-98) [4f0c66c3] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS >2014-07-27 10:43:06,975 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-98) [71c44c7e] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-07-27 10:43:06,984 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-98) [5909f95c] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-07-27 10:43:06,996 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-98) [5909f95c] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, storageType = NFS, connectionList = [{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 6235aea6 >2014-07-27 10:43:07,052 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-98) [5909f95c] FINISH, ConnectStorageServerVDSCommand, return: {5083f370-af09-457b-9b73-3440e8b65a07=0}, log id: 6235aea6 >2014-07-27 10:43:07,054 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-98) [5909f95c] Host rack storage connection was succeeded >2014-07-27 10:43:07,062 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-17) START, ConnectStoragePoolVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, masterVersion = 1), log id: 465ca7d8 >2014-07-27 10:43:13,300 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-3) Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Inactive to Unknown, reason : no reporting hosts >2014-07-27 10:44:11,535 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-17) FINISH, ConnectStoragePoolVDSCommand, log id: 465ca7d8 >2014-07-27 10:44:11,548 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-98) [5909f95c] START, SetMOMPolicyParametersVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 7a6c2703 >2014-07-27 10:44:11,641 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-98) [5909f95c] FINISH, SetMOMPolicyParametersVDSCommand, log id: 7a6c2703 >2014-07-27 10:44:11,648 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-98) [5909f95c] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host rack was set to Up. >2014-07-27 10:44:11,654 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-98) [5909f95c] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host rack. >2014-07-27 10:44:11,670 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-98) [5aa74089] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS >2014-07-27 10:44:13,857 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-42) hostFromVds::selectedVds - rack, spmStatus Free, storage pool Default >2014-07-27 10:44:13,863 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-42) starting spm on vds rack, storage pool Default, prevId -1, LVER -1 >2014-07-27 10:44:13,865 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-42) START, SpmStartVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, prevId=-1, prevLVER=-1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 3905a945 >2014-07-27 10:44:13,913 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-42) spmStart polling started: taskId = 0d169a53-0a63-48cd-9d32-55c245254da1 >2014-07-27 10:44:15,924 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-42) spmStart polling ended: taskId = 0d169a53-0a63-48cd-9d32-55c245254da1 task status = finished >2014-07-27 10:44:15,982 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-42) spmStart polling ended, spm status: SPM >2014-07-27 10:44:15,984 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-42) START, HSMClearTaskVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, taskId=0d169a53-0a63-48cd-9d32-55c245254da1), log id: 57836cb1 >2014-07-27 10:44:16,029 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-42) FINISH, HSMClearTaskVDSCommand, log id: 57836cb1 >2014-07-27 10:44:16,030 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-42) FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@63f08866, log id: 3905a945 >2014-07-27 10:44:16,034 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-42) Initialize Irs proxy from vds: 10.34.62.208 >2014-07-27 10:44:16,042 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-42) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208). >2014-07-27 10:44:16,043 WARN [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-8-thread-3) Executing a command: java.util.concurrent.FutureTask , but note that there are 1 tasks in the queue. >2014-07-27 10:44:16,056 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-3) START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 5a9a3132 >2014-07-27 10:44:16,080 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-3) -- executeIrsBrokerCommand: Attempting on storage pool 00000002-0002-0002-0002-0000000001d9 >2014-07-27 10:44:16,082 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-3) START, HSMGetAllTasksInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 47cdeb08 >2014-07-27 10:44:16,087 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-3) FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 47cdeb08 >2014-07-27 10:44:16,088 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-3) FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 5a9a3132 >2014-07-27 10:44:16,089 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (org.ovirt.thread.pool-8-thread-3) Discovered no tasks on Storage Pool Default >2014-07-27 11:23:54,628 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-55) Principal admin::internal synchronized >2014-07-27 11:24:00,138 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-32) Attempting to update VMs/Templates Ovf. >2014-07-27 12:23:54,636 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-5) Principal admin::internal synchronized >2014-07-27 12:24:00,170 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-1) Attempting to update VMs/Templates Ovf. >2014-07-27 13:23:54,642 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-38) [1b9ef0f] Principal admin::internal synchronized >2014-07-27 13:24:00,201 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-4) Attempting to update VMs/Templates Ovf. >2014-07-27 14:23:54,648 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-3) Principal admin::internal synchronized >2014-07-27 14:24:00,213 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-65) Attempting to update VMs/Templates Ovf. >2014-07-27 14:55:01,130 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-53) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage domain 35 experienced a high latency of 6.1082 seconds from host rack. This may cause performance and functional issues. Please consult your Storage Administrator. >2014-07-27 15:23:54,653 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-61) [58bfe349] Principal admin::internal synchronized >2014-07-27 15:24:00,226 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-66) [2734597c] Attempting to update VMs/Templates Ovf. >2014-07-27 16:23:54,670 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-97) Principal admin::internal synchronized >2014-07-27 16:24:00,241 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-40) Attempting to update VMs/Templates Ovf. >2014-07-27 16:31:05,625 WARN [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-8-thread-16) Executing a command: java.util.concurrent.FutureTask , but note that there are 1 tasks in the queue. >2014-07-27 17:23:54,678 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-43) Principal admin::internal synchronized >2014-07-27 17:24:00,272 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-31) Attempting to update VMs/Templates Ovf. >2014-07-27 18:23:54,683 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-96) Principal admin::internal synchronized >2014-07-27 18:24:00,286 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-18) Attempting to update VMs/Templates Ovf. >2014-07-27 18:30:00,449 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-24) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage domain 35 experienced a high latency of 6.15416 seconds from host rack. This may cause performance and functional issues. Please consult your Storage Administrator. >2014-07-27 19:05:02,358 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-43) domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 in problem. vds: rack >2014-07-27 19:05:47,713 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-15) Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 recovered from problem. vds: rack >2014-07-27 19:05:47,714 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-15) Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 has recovered from problem. No active host in the DC is reporting it as problematic, so clearing the domain recovery timer. >2014-07-27 19:23:54,689 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-34) Principal admin::internal synchronized >2014-07-27 19:24:00,304 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-14) Attempting to update VMs/Templates Ovf. >2014-07-27 20:07:02,563 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-47) domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 in problem. vds: rack >2014-07-27 20:08:30,114 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-75) Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2014-07-27 20:08:30,119 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-75) [52dee25c] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-07-27 20:08:30,122 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-75) [52dee25c] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Active to Unknown, reason : null >2014-07-27 20:08:30,134 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-75) [52dee25c] Correlation ID: 52dee25c, Call Stack: null, Custom Event ID: -1, Message: Invalid status on Data Center Default. Setting Data Center status to Non Responsive (On host rack, Error: Cannot access master storage domain). >2014-07-27 20:10:30,163 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-75) [52dee25c] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2014-07-27 20:10:30,165 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-75) [52dee25c] hostFromVds::selectedVds - rack, spmStatus returned null! >2014-07-27 20:10:30,167 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-75) [52dee25c] IrsBroker::Failed::GetStoragePoolInfoVDS >2014-07-27 20:10:30,168 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-75) [52dee25c] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2014-07-27 20:10:30,200 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-18) [40b4906d] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage >2014-07-27 20:10:30,217 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-18) [40b4906d] START, ResetIrsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, ignoreStopFailed = true), log id: 27e1459d >2014-07-27 20:10:30,221 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-18) [40b4906d] START, SpmStopVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9), log id: 768443a3 >2014-07-27 20:10:30,227 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-18) [40b4906d] SpmStopVDSCommand::Stopping SPM on vds rack, pool id 00000002-0002-0002-0002-0000000001d9 >2014-07-27 20:12:30,134 WARN [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-18) [40b4906d] could not stop spm of pool 00000002-0002-0002-0002-0000000001d9 on vds 17f7fe56-5f32-40ef-b098-611b84484e0b - reason: org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcRunTimeException: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException >2014-07-27 20:12:30,138 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-18) [40b4906d] FINISH, SpmStopVDSCommand, log id: 768443a3 >2014-07-27 20:12:30,149 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-2) Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: VDSNetworkException: org.apache.commons.httpclient.NoHttpResponseException: The server 10.34.62.208 failed to respond >2014-07-27 20:12:30,151 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-2) hostFromVds::selectedVds - rack, spmStatus returned null! >2014-07-27 20:12:30,153 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-18) [40b4906d] FINISH, ResetIrsVDSCommand, log id: 27e1459d >2014-07-27 20:12:30,157 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ListVDSCommand] (DefaultQuartzScheduler_Worker-77) Command ListVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-07-27 20:12:30,162 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-77) Host rack is not responding. It will stay in Connecting state for a grace period of $80 seconds and after that an attempt to fence the host will be issued. >2014-07-27 20:12:30,163 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-77) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:406) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.fetchRunningVms(VdsUpdateRunTimeInfo.java:971) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVmStats(VdsUpdateRunTimeInfo.java:920) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:638) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:235) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) [:1.7.0_55] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_55] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_55] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_55] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_55] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_55] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_55] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_55] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy82.list(Unknown Source) > at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source) [:1.7.0_55] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_55] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_55] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_55] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_55] > >2014-07-27 20:12:30,167 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-18) [40b4906d] Correlation ID: 40b4906d, Job ID: b3f62e68-097e-4c0b-b35a-0124dfe278b4, Call Stack: null, Custom Event ID: -1, Message: Failed to Reconstruct Master Domain for Data Center Default. >2014-07-27 20:12:30,202 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-1) [52dee25c] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue >2014-07-27 20:12:30,205 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-5) starting processDomainRecovery for domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 >2014-07-27 20:12:30,206 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-77) Failed to refresh VDS , vds = 17f7fe56-5f32-40ef-b098-611b84484e0b : rack, VDS Network Error, continuing. >java.net.ConnectException: Connection refused >2014-07-27 20:12:30,274 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-5) Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 was reported by all hosts in status UP as problematic. Not moving the domain to NonOperational because it is being reconstructed now. >2014-07-27 20:12:30,303 WARN [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-5) [57c73f48] CanDoAction of action ReconstructMasterDomain failed. Reasons:VAR__ACTION__RECONSTRUCT_MASTER,VAR__TYPE__STORAGE__DOMAIN,ACTION_TYPE_FAILED_NO_VDS_IN_POOL >2014-07-27 20:12:30,312 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-5) [57c73f48] Removing vds [] from the domain in maintenance cache >2014-07-27 20:12:30,313 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-5) [57c73f48] Removing host(s) [] from hosts unseen domain report cache >2014-07-27 20:12:30,314 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-1) [52dee25c] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue >2014-07-27 20:12:33,295 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-48) Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@46ea165b >2014-07-27 20:12:33,297 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-48) HostName = rack >2014-07-27 20:12:33,297 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-48) Command GetCapabilitiesVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]) execution failed. Exception: VDSRecoveringException: Recovering from crash or Initializing >2014-07-27 20:12:33,324 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-48) Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSRecoveringException: Recovering from crash or Initializing > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:16) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:566) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:235) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Host rack is initializing. Message: Recovering from crash or Initializing >2014-07-27 20:12:33,331 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-48) Failed to refresh VDS , vds = 17f7fe56-5f32-40ef-b098-611b84484e0b : rack, error = Recovering from crash or Initializing, continuing. >2014-07-27 20:12:36,572 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-100) START, GetHardwareInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]), log id: 78c1c62b >2014-07-27 20:12:36,626 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-100) FINISH, GetHardwareInfoVDSCommand, log id: 78c1c62b >2014-07-27 20:12:36,670 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-100) [1df66af2] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS >2014-07-27 20:12:36,679 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-100) [1df66af2] Cancelling the recovery from crash timer for VDS 17f7fe56-5f32-40ef-b098-611b84484e0b because vds started initializing >2014-07-27 20:12:36,682 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-100) [685f7cc1] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-07-27 20:12:36,693 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-100) [5dd5c35c] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-07-27 20:12:36,705 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-100) [5dd5c35c] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, storageType = NFS, connectionList = [{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: d4ff2cf >2014-07-27 20:12:36,762 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-100) [5dd5c35c] FINISH, ConnectStorageServerVDSCommand, return: {5083f370-af09-457b-9b73-3440e8b65a07=0}, log id: d4ff2cf >2014-07-27 20:12:36,763 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-100) [5dd5c35c] Host rack storage connection was succeeded >2014-07-27 20:12:36,772 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-31) START, ConnectStoragePoolVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, masterVersion = 1), log id: 227014c >2014-07-27 20:12:40,161 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-76) Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Inactive to Unknown, reason : no reporting hosts >2014-07-27 20:13:37,215 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-31) FINISH, ConnectStoragePoolVDSCommand, log id: 227014c >2014-07-27 20:13:37,229 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-100) [5dd5c35c] START, SetMOMPolicyParametersVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 579b07b1 >2014-07-27 20:13:37,323 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-100) [5dd5c35c] FINISH, SetMOMPolicyParametersVDSCommand, log id: 579b07b1 >2014-07-27 20:13:37,330 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-100) [5dd5c35c] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host rack was set to Up. >2014-07-27 20:13:37,337 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-100) [5dd5c35c] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host rack. >2014-07-27 20:13:37,356 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-100) [12a64afb] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS >2014-07-27 20:13:40,274 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-81) hostFromVds::selectedVds - rack, spmStatus Free, storage pool Default >2014-07-27 20:13:40,279 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-81) starting spm on vds rack, storage pool Default, prevId -1, LVER -1 >2014-07-27 20:13:40,281 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-81) START, SpmStartVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, prevId=-1, prevLVER=-1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 720c77db >2014-07-27 20:13:40,331 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-81) spmStart polling started: taskId = dee365f4-3ae8-417e-93b4-914603a97a8c >2014-07-27 20:13:41,337 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-81) spmStart polling ended: taskId = dee365f4-3ae8-417e-93b4-914603a97a8c task status = finished >2014-07-27 20:13:41,392 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-81) spmStart polling ended, spm status: SPM >2014-07-27 20:13:41,395 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-81) START, HSMClearTaskVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, taskId=dee365f4-3ae8-417e-93b4-914603a97a8c), log id: 5dedde83 >2014-07-27 20:13:41,441 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-81) FINISH, HSMClearTaskVDSCommand, log id: 5dedde83 >2014-07-27 20:13:41,442 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-81) FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@3b28499b, log id: 720c77db >2014-07-27 20:13:41,446 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-81) Initialize Irs proxy from vds: 10.34.62.208 >2014-07-27 20:13:41,451 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-81) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208). >2014-07-27 20:13:41,461 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-20) START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 1623a10b >2014-07-27 20:13:41,490 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-20) -- executeIrsBrokerCommand: Attempting on storage pool 00000002-0002-0002-0002-0000000001d9 >2014-07-27 20:13:41,491 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-20) START, HSMGetAllTasksInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 1666e8a4 >2014-07-27 20:13:41,497 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-20) FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 1666e8a4 >2014-07-27 20:13:41,497 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-20) FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 1623a10b >2014-07-27 20:13:41,498 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (org.ovirt.thread.pool-8-thread-20) Discovered no tasks on Storage Pool Default >2014-07-27 20:23:54,696 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-3) Principal admin::internal synchronized >2014-07-27 20:24:00,320 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-45) Attempting to update VMs/Templates Ovf. >2014-07-27 20:45:41,027 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-28) domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 in problem. vds: rack >2014-07-27 20:45:56,151 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-5) Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 recovered from problem. vds: rack >2014-07-27 20:45:56,152 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-5) Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 has recovered from problem. No active host in the DC is reporting it as problematic, so clearing the domain recovery timer. >2014-07-27 21:23:54,703 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-96) Principal admin::internal synchronized >2014-07-27 21:24:00,336 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-29) Attempting to update VMs/Templates Ovf. >2014-07-27 21:28:10,510 WARN [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-8-thread-22) Executing a command: java.util.concurrent.FutureTask , but note that there are 1 tasks in the queue. >2014-07-27 22:23:54,712 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-38) [1b9ef0f] Principal admin::internal synchronized >2014-07-27 22:24:00,367 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-90) Attempting to update VMs/Templates Ovf. >2014-07-27 23:23:54,718 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-52) Principal admin::internal synchronized >2014-07-27 23:24:00,380 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-1) Attempting to update VMs/Templates Ovf. >2014-07-28 00:23:54,734 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-27) Principal admin::internal synchronized >2014-07-28 00:24:00,393 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-36) [3e2ffa8] Attempting to update VMs/Templates Ovf. >2014-07-28 01:23:54,739 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-66) [2734597c] Principal admin::internal synchronized >2014-07-28 01:24:00,409 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-56) Attempting to update VMs/Templates Ovf. >2014-07-28 02:23:54,746 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-83) Principal admin::internal synchronized >2014-07-28 02:24:00,421 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-47) Attempting to update VMs/Templates Ovf. >2014-07-28 03:23:54,750 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-17) Principal admin::internal synchronized >2014-07-28 03:24:00,434 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-69) Attempting to update VMs/Templates Ovf. >2014-07-28 03:35:35,001 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-11) Start deleteAgedOutAuditLogs >2014-07-28 03:35:35,001 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-33) Start deleteAgedOutCommandEntities >2014-07-28 03:35:35,003 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-11) Finished deleteAgedOutAuditLogs >2014-07-28 03:35:35,009 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-33) Finished deleteAgedOutCommandEntities >2014-07-28 04:23:54,758 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-26) Principal admin::internal synchronized >2014-07-28 04:24:00,448 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-77) Attempting to update VMs/Templates Ovf. >2014-07-28 05:23:54,764 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-94) Principal admin::internal synchronized >2014-07-28 05:24:00,495 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-87) Attempting to update VMs/Templates Ovf. >2014-07-28 05:24:53,022 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-25) domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 in problem. vds: rack >2014-07-28 05:26:22,067 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-95) Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2014-07-28 05:26:22,072 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-95) [75d6be08] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-07-28 05:26:22,075 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-95) [75d6be08] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Active to Unknown, reason : null >2014-07-28 05:26:22,108 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-95) [75d6be08] Correlation ID: 75d6be08, Call Stack: null, Custom Event ID: -1, Message: Invalid status on Data Center Default. Setting Data Center status to Non Responsive (On host rack, Error: Cannot access master storage domain). >2014-07-28 05:28:22,118 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-95) [75d6be08] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2014-07-28 05:28:22,120 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-95) [75d6be08] hostFromVds::selectedVds - rack, spmStatus returned null! >2014-07-28 05:28:22,121 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-95) [75d6be08] IrsBroker::Failed::GetStoragePoolInfoVDS >2014-07-28 05:28:22,122 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-95) [75d6be08] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2014-07-28 05:28:22,163 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-16) [44b05f41] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage >2014-07-28 05:28:22,180 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-16) [44b05f41] START, ResetIrsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, ignoreStopFailed = true), log id: 4baa9eba >2014-07-28 05:28:22,183 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-16) [44b05f41] START, SpmStopVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9), log id: 69a518e5 >2014-07-28 05:28:22,189 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-16) [44b05f41] SpmStopVDSCommand::Stopping SPM on vds rack, pool id 00000002-0002-0002-0002-0000000001d9 >2014-07-28 05:30:00,005 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-29) Autorecovering 1 storage domains >2014-07-28 05:30:00,007 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-29) Autorecovering storage domains id: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 >2014-07-28 05:30:00,010 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-29) [327b509d] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage >2014-07-28 05:30:00,013 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-29) [327b509d] ConnectDomainToStorage. Before Connect all hosts to pool. Time:7/28/14 5:30 AM >2014-07-28 05:30:00,024 INFO [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-8-thread-36) [3c16f387] Running command: ConnectStorageToVdsCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2014-07-28 05:30:00,030 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-36) [3c16f387] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 497b3dd >2014-07-28 05:30:22,097 WARN [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-16) [44b05f41] could not stop spm of pool 00000002-0002-0002-0002-0000000001d9 on vds 17f7fe56-5f32-40ef-b098-611b84484e0b - reason: org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcRunTimeException: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException >2014-07-28 05:30:22,099 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-16) [44b05f41] FINISH, SpmStopVDSCommand, log id: 69a518e5 >2014-07-28 05:30:22,103 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-36) [3c16f387] Command ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-07-28 05:30:22,107 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-44) [60dd31b0] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: VDSNetworkException: org.apache.commons.httpclient.NoHttpResponseException: The server 10.34.62.208 failed to respond >2014-07-28 05:30:22,104 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ListVDSCommand] (DefaultQuartzScheduler_Worker-54) Command ListVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-07-28 05:30:22,114 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-44) [60dd31b0] hostFromVds::selectedVds - rack, spmStatus returned null! >2014-07-28 05:30:22,114 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-36) [3c16f387] FINISH, ConnectStorageServerVDSCommand, log id: 497b3dd >2014-07-28 05:30:22,118 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-16) [44b05f41] FINISH, ResetIrsVDSCommand, log id: 4baa9eba >2014-07-28 05:30:22,116 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-54) Host rack is not responding. It will stay in Connecting state for a grace period of $80 seconds and after that an attempt to fence the host will be issued. >2014-07-28 05:30:22,118 ERROR [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-8-thread-36) [3c16f387] Command org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-07-28 05:30:22,120 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-54) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:406) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.fetchRunningVms(VdsUpdateRunTimeInfo.java:971) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVmStats(VdsUpdateRunTimeInfo.java:920) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:638) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:235) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) [:1.7.0_55] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_55] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_55] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_55] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_55] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_55] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_55] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_55] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy82.list(Unknown Source) > at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source) [:1.7.0_55] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_55] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_55] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_55] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_55] > >2014-07-28 05:30:22,134 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-16) [44b05f41] Correlation ID: 44b05f41, Job ID: a958ccca-6f65-4e38-b975-60c971a1630c, Call Stack: null, Custom Event ID: -1, Message: Failed to Reconstruct Master Domain for Data Center Default. >2014-07-28 05:30:22,134 ERROR [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-8-thread-36) [3c16f387] Transaction rolled-back for command: org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand. >2014-07-28 05:30:22,156 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-54) Failed to refresh VDS , vds = 17f7fe56-5f32-40ef-b098-611b84484e0b : rack, VDS Network Error, continuing. >java.net.ConnectException: Connection refused >2014-07-28 05:30:22,158 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-29) [327b509d] ConnectDomainToStorage. After Connect all hosts to pool. Time:7/28/14 5:30 AM >2014-07-28 05:30:22,160 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-3) [75d6be08] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue >2014-07-28 05:30:22,162 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-12) starting processDomainRecovery for domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 >2014-07-28 05:30:22,249 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-12) Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 was reported by all hosts in status UP as problematic. Not moving the domain to NonOperational because it is being reconstructed now. >2014-07-28 05:30:22,276 WARN [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-12) [1892618e] CanDoAction of action ReconstructMasterDomain failed. Reasons:VAR__ACTION__RECONSTRUCT_MASTER,VAR__TYPE__STORAGE__DOMAIN,ACTION_TYPE_FAILED_NO_VDS_IN_POOL >2014-07-28 05:30:22,284 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-12) [1892618e] Removing vds [] from the domain in maintenance cache >2014-07-28 05:30:22,285 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-12) [1892618e] Removing host(s) [] from hosts unseen domain report cache >2014-07-28 05:30:22,286 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-3) [75d6be08] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue >2014-07-28 05:30:25,240 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-25) Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@27f19f5b >2014-07-28 05:30:25,242 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-25) HostName = rack >2014-07-28 05:30:25,243 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-25) Command GetCapabilitiesVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]) execution failed. Exception: VDSRecoveringException: Recovering from crash or Initializing >2014-07-28 05:30:25,256 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-25) Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSRecoveringException: Recovering from crash or Initializing > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:16) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:566) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:235) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Host rack is initializing. Message: Recovering from crash or Initializing >2014-07-28 05:30:25,263 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-25) Failed to refresh VDS , vds = 17f7fe56-5f32-40ef-b098-611b84484e0b : rack, error = Recovering from crash or Initializing, continuing. >2014-07-28 05:30:28,276 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-8) Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@10ac4b2c >2014-07-28 05:30:28,278 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-8) HostName = rack >2014-07-28 05:30:28,279 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-8) Command GetCapabilitiesVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]) execution failed. Exception: VDSRecoveringException: Recovering from crash or Initializing >2014-07-28 05:30:31,509 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-39) START, GetHardwareInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]), log id: 10939650 >2014-07-28 05:30:31,560 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-39) FINISH, GetHardwareInfoVDSCommand, log id: 10939650 >2014-07-28 05:30:31,585 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-39) [7f2483a0] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS >2014-07-28 05:30:31,636 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-39) [7f2483a0] Cancelling the recovery from crash timer for VDS 17f7fe56-5f32-40ef-b098-611b84484e0b because vds started initializing >2014-07-28 05:30:31,638 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-39) [4beb5dac] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-07-28 05:30:31,646 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-39) [fdf6fa5] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-07-28 05:30:31,658 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-39) [fdf6fa5] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, storageType = NFS, connectionList = [{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 6cd4bf1e >2014-07-28 05:30:31,714 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-39) [fdf6fa5] FINISH, ConnectStorageServerVDSCommand, return: {5083f370-af09-457b-9b73-3440e8b65a07=0}, log id: 6cd4bf1e >2014-07-28 05:30:31,716 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-39) [fdf6fa5] Host rack storage connection was succeeded >2014-07-28 05:30:31,723 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-28) START, ConnectStoragePoolVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, masterVersion = 1), log id: 2bc4b16 >2014-07-28 05:30:32,122 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-79) Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Inactive to Unknown, reason : no reporting hosts >2014-07-28 05:31:38,719 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-28) FINISH, ConnectStoragePoolVDSCommand, log id: 2bc4b16 >2014-07-28 05:31:38,731 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-39) [fdf6fa5] START, SetMOMPolicyParametersVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 767366e7 >2014-07-28 05:31:38,824 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-39) [fdf6fa5] FINISH, SetMOMPolicyParametersVDSCommand, log id: 767366e7 >2014-07-28 05:31:38,831 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-39) [fdf6fa5] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host rack was set to Up. >2014-07-28 05:31:38,837 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-39) [fdf6fa5] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host rack. >2014-07-28 05:31:38,856 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-39) [4a2f7e33] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS >2014-07-28 05:31:42,570 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-45) hostFromVds::selectedVds - rack, spmStatus Free, storage pool Default >2014-07-28 05:31:42,578 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-45) starting spm on vds rack, storage pool Default, prevId -1, LVER -1 >2014-07-28 05:31:42,582 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-45) START, SpmStartVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, prevId=-1, prevLVER=-1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 3474374 >2014-07-28 05:31:42,633 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-45) spmStart polling started: taskId = 16f6da0a-6893-4a05-acb1-d3a4342979ac >2014-07-28 05:31:46,666 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-45) spmStart polling ended: taskId = 16f6da0a-6893-4a05-acb1-d3a4342979ac task status = finished >2014-07-28 05:31:46,734 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-45) spmStart polling ended, spm status: SPM >2014-07-28 05:31:46,737 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-45) START, HSMClearTaskVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, taskId=16f6da0a-6893-4a05-acb1-d3a4342979ac), log id: 2a5c481b >2014-07-28 05:31:46,782 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-45) FINISH, HSMClearTaskVDSCommand, log id: 2a5c481b >2014-07-28 05:31:46,784 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-45) FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@a246513, log id: 3474374 >2014-07-28 05:31:46,798 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-45) Initialize Irs proxy from vds: 10.34.62.208 >2014-07-28 05:31:46,807 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-45) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208). >2014-07-28 05:31:46,809 WARN [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-8-thread-11) Executing a command: java.util.concurrent.FutureTask , but note that there are 1 tasks in the queue. >2014-07-28 05:31:46,820 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-11) START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 65e5fbdf >2014-07-28 05:31:47,132 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-11) -- executeIrsBrokerCommand: Attempting on storage pool 00000002-0002-0002-0002-0000000001d9 >2014-07-28 05:31:47,135 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-11) START, HSMGetAllTasksInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 957fbd5 >2014-07-28 05:31:47,141 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-11) FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 957fbd5 >2014-07-28 05:31:47,142 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-11) FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 65e5fbdf >2014-07-28 05:31:47,144 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (org.ovirt.thread.pool-8-thread-11) Discovered no tasks on Storage Pool Default >2014-07-28 06:23:54,772 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-70) Principal admin::internal synchronized >2014-07-28 06:24:00,510 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-36) [3e2ffa8] Attempting to update VMs/Templates Ovf. >2014-07-28 07:23:54,780 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-85) Principal admin::internal synchronized >2014-07-28 07:24:00,523 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-87) Attempting to update VMs/Templates Ovf. >2014-07-28 08:23:54,786 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-15) Principal admin::internal synchronized >2014-07-28 08:24:00,542 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-40) Attempting to update VMs/Templates Ovf. >2014-07-28 09:23:54,792 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-51) Principal admin::internal synchronized >2014-07-28 09:24:00,556 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-36) [3e2ffa8] Attempting to update VMs/Templates Ovf. >2014-07-28 10:23:54,797 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-21) Principal admin::internal synchronized >2014-07-28 10:24:00,572 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-78) Attempting to update VMs/Templates Ovf. >2014-07-28 11:23:54,803 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-60) Principal admin::internal synchronized >2014-07-28 11:24:00,588 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-27) Attempting to update VMs/Templates Ovf. >2014-07-28 12:23:54,810 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-66) [2734597c] Principal admin::internal synchronized >2014-07-28 12:24:00,624 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-97) Attempting to update VMs/Templates Ovf. >2014-07-28 13:23:54,817 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-41) Principal admin::internal synchronized >2014-07-28 13:24:00,655 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-80) Attempting to update VMs/Templates Ovf. >2014-07-28 14:23:54,822 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-54) Principal admin::internal synchronized >2014-07-28 14:24:00,670 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-89) Attempting to update VMs/Templates Ovf. >2014-07-28 14:53:39,769 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-45) domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 in problem. vds: rack >2014-07-28 14:55:14,710 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-67) Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2014-07-28 14:55:14,717 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-67) [2f0280a3] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-07-28 14:55:14,722 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-67) [2f0280a3] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Active to Unknown, reason : null >2014-07-28 14:55:14,735 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-67) [2f0280a3] Correlation ID: 2f0280a3, Call Stack: null, Custom Event ID: -1, Message: Invalid status on Data Center Default. Setting Data Center status to Non Responsive (On host rack, Error: Cannot access master storage domain). >2014-07-28 14:57:14,760 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-67) [2f0280a3] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2014-07-28 14:57:14,762 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-67) [2f0280a3] hostFromVds::selectedVds - rack, spmStatus returned null! >2014-07-28 14:57:14,763 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-67) [2f0280a3] IrsBroker::Failed::GetStoragePoolInfoVDS >2014-07-28 14:57:14,764 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-67) [2f0280a3] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2014-07-28 14:57:14,795 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-44) [59aadc32] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage >2014-07-28 14:57:14,813 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-44) [59aadc32] START, ResetIrsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, ignoreStopFailed = true), log id: 5ca7f0a4 >2014-07-28 14:57:14,817 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-44) [59aadc32] START, SpmStopVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9), log id: 948523b >2014-07-28 14:57:14,823 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-44) [59aadc32] SpmStopVDSCommand::Stopping SPM on vds rack, pool id 00000002-0002-0002-0002-0000000001d9 >2014-07-28 14:59:14,729 WARN [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-44) [59aadc32] could not stop spm of pool 00000002-0002-0002-0002-0000000001d9 on vds 17f7fe56-5f32-40ef-b098-611b84484e0b - reason: org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcRunTimeException: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException >2014-07-28 14:59:14,732 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-44) [59aadc32] FINISH, SpmStopVDSCommand, log id: 948523b >2014-07-28 14:59:14,735 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-5) Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: VDSNetworkException: org.apache.commons.httpclient.NoHttpResponseException: The server 10.34.62.208 failed to respond >2014-07-28 14:59:14,739 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetStatsVDSCommand] (DefaultQuartzScheduler_Worker-2) Command GetStatsVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-07-28 14:59:14,742 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-44) [59aadc32] FINISH, ResetIrsVDSCommand, log id: 5ca7f0a4 >2014-07-28 14:59:14,740 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-5) hostFromVds::selectedVds - rack, spmStatus returned null! >2014-07-28 14:59:14,744 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-2) Host rack is not responding. It will stay in Connecting state for a grace period of $80 seconds and after that an attempt to fence the host will be issued. >2014-07-28 14:59:14,748 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-2) vds::refreshVdsStats Failed getVdsStats, vds = 17f7fe56-5f32-40ef-b098-611b84484e0b : rack, error = VDSNetworkException: java.net.ConnectException: Connection refused >2014-07-28 14:59:14,752 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-2) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsStats(VdsUpdateRunTimeInfo.java:669) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:623) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:235) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) [:1.7.0_55] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_55] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_55] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_55] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_55] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_55] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_55] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_55] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy82.getVdsStats(Unknown Source) > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_55] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_55] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_55] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_55] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_55] > >2014-07-28 14:59:14,765 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-44) [59aadc32] Correlation ID: 59aadc32, Job ID: ef052d9e-7451-4f4b-b3d1-b767dad6b2e5, Call Stack: null, Custom Event ID: -1, Message: Failed to Reconstruct Master Domain for Data Center Default. >2014-07-28 14:59:14,793 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-2) Failed to refresh VDS , vds = 17f7fe56-5f32-40ef-b098-611b84484e0b : rack, VDS Network Error, continuing. >java.net.ConnectException: Connection refused >2014-07-28 14:59:14,797 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-29) [2f0280a3] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue >2014-07-28 14:59:14,799 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-24) starting processDomainRecovery for domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 >2014-07-28 14:59:14,865 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-24) Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 was reported by all hosts in status UP as problematic. Not moving the domain to NonOperational because it is being reconstructed now. >2014-07-28 14:59:14,894 WARN [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-24) [62148fba] CanDoAction of action ReconstructMasterDomain failed. Reasons:VAR__ACTION__RECONSTRUCT_MASTER,VAR__TYPE__STORAGE__DOMAIN,ACTION_TYPE_FAILED_NO_VDS_IN_POOL >2014-07-28 14:59:14,902 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-24) [62148fba] Removing vds [] from the domain in maintenance cache >2014-07-28 14:59:14,903 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-24) [62148fba] Removing host(s) [] from hosts unseen domain report cache >2014-07-28 14:59:14,904 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-29) [2f0280a3] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue >2014-07-28 14:59:18,051 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-84) START, GetHardwareInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]), log id: 510a2203 >2014-07-28 14:59:18,106 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-84) FINISH, GetHardwareInfoVDSCommand, log id: 510a2203 >2014-07-28 14:59:18,129 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-84) [3af88bf7] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS >2014-07-28 14:59:18,178 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-84) [25ea9bee] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-07-28 14:59:18,186 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-84) [c3bdd3d] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-07-28 14:59:18,194 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-84) [c3bdd3d] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, storageType = NFS, connectionList = [{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 674689c4 >2014-07-28 14:59:18,247 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-84) [c3bdd3d] FINISH, ConnectStorageServerVDSCommand, return: {5083f370-af09-457b-9b73-3440e8b65a07=0}, log id: 674689c4 >2014-07-28 14:59:18,248 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-84) [c3bdd3d] Host rack storage connection was succeeded >2014-07-28 14:59:18,254 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-12) START, ConnectStoragePoolVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, masterVersion = 1), log id: 71f70b65 >2014-07-28 14:59:24,750 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-45) Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Inactive to Unknown, reason : no reporting hosts >2014-07-28 15:00:18,616 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-12) FINISH, ConnectStoragePoolVDSCommand, log id: 71f70b65 >2014-07-28 15:00:18,628 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-84) [c3bdd3d] START, SetMOMPolicyParametersVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 10685ea8 >2014-07-28 15:00:18,721 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-84) [c3bdd3d] FINISH, SetMOMPolicyParametersVDSCommand, log id: 10685ea8 >2014-07-28 15:00:18,731 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-84) [c3bdd3d] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host rack was set to Up. >2014-07-28 15:00:18,738 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-84) [c3bdd3d] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host rack. >2014-07-28 15:00:18,754 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-84) [7ef2d911] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS >2014-07-28 15:00:24,851 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-55) hostFromVds::selectedVds - rack, spmStatus Free, storage pool Default >2014-07-28 15:00:24,857 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-55) starting spm on vds rack, storage pool Default, prevId -1, LVER -1 >2014-07-28 15:00:24,860 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-55) START, SpmStartVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, prevId=-1, prevLVER=-1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 34a8334f >2014-07-28 15:00:24,908 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-55) spmStart polling started: taskId = a794c812-083e-47c3-a1c0-972895bb6e6b >2014-07-28 15:00:25,916 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-55) spmStart polling ended: taskId = a794c812-083e-47c3-a1c0-972895bb6e6b task status = finished >2014-07-28 15:00:25,971 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-55) spmStart polling ended, spm status: SPM >2014-07-28 15:00:25,974 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-55) START, HSMClearTaskVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, taskId=a794c812-083e-47c3-a1c0-972895bb6e6b), log id: 403883bd >2014-07-28 15:00:26,018 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-55) FINISH, HSMClearTaskVDSCommand, log id: 403883bd >2014-07-28 15:00:26,019 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-55) FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@4387eab8, log id: 34a8334f >2014-07-28 15:00:26,023 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-55) Initialize Irs proxy from vds: 10.34.62.208 >2014-07-28 15:00:26,029 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-55) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208). >2014-07-28 15:00:26,045 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-35) START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 36d29e43 >2014-07-28 15:00:26,069 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-35) -- executeIrsBrokerCommand: Attempting on storage pool 00000002-0002-0002-0002-0000000001d9 >2014-07-28 15:00:26,071 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-35) START, HSMGetAllTasksInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 1bf40fba >2014-07-28 15:00:26,076 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-35) FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 1bf40fba >2014-07-28 15:00:26,077 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-35) FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 36d29e43 >2014-07-28 15:00:26,078 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (org.ovirt.thread.pool-8-thread-35) Discovered no tasks on Storage Pool Default >2014-07-28 15:23:54,827 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-70) Principal admin::internal synchronized >2014-07-28 15:24:00,700 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-38) [1b9ef0f] Attempting to update VMs/Templates Ovf. >2014-07-28 16:23:54,831 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-79) Principal admin::internal synchronized >2014-07-28 16:24:00,715 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-72) Attempting to update VMs/Templates Ovf. >2014-07-28 16:30:59,760 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-47) domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 in problem. vds: rack >2014-07-28 16:31:14,920 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-44) Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 recovered from problem. vds: rack >2014-07-28 16:31:14,922 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-44) Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 has recovered from problem. No active host in the DC is reporting it as problematic, so clearing the domain recovery timer. >2014-07-28 17:23:54,836 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-73) Principal admin::internal synchronized >2014-07-28 17:24:00,729 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-97) Attempting to update VMs/Templates Ovf. >2014-07-28 18:23:54,844 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-39) [4a2f7e33] Principal admin::internal synchronized >2014-07-28 18:24:00,743 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-77) Attempting to update VMs/Templates Ovf. >2014-07-28 19:23:54,851 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-79) Principal admin::internal synchronized >2014-07-28 19:24:00,780 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-25) Attempting to update VMs/Templates Ovf. >2014-07-28 20:23:54,857 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-65) Principal admin::internal synchronized >2014-07-28 20:24:00,794 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-23) Attempting to update VMs/Templates Ovf. >2014-07-28 21:00:31,448 WARN [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-8-thread-10) Executing a command: java.util.concurrent.FutureTask , but note that there are 1 tasks in the queue. >2014-07-28 21:23:54,863 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-14) Principal admin::internal synchronized >2014-07-28 21:24:00,811 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-95) [75d6be08] Attempting to update VMs/Templates Ovf. >2014-07-28 22:23:54,872 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-59) Principal admin::internal synchronized >2014-07-28 22:24:00,823 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-94) Attempting to update VMs/Templates Ovf. >2014-07-28 23:23:54,878 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-65) Principal admin::internal synchronized >2014-07-28 23:24:00,852 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-1) Attempting to update VMs/Templates Ovf. >2014-07-29 00:23:18,509 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-13) domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 in problem. vds: rack >2014-07-29 00:23:54,885 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-90) Principal admin::internal synchronized >2014-07-29 00:24:00,899 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-11) Attempting to update VMs/Templates Ovf. >2014-07-29 00:24:50,899 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-56) Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2014-07-29 00:24:50,903 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-56) [738ebb7b] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-07-29 00:24:50,907 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-56) [738ebb7b] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Active to Unknown, reason : null >2014-07-29 00:24:50,920 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-56) [738ebb7b] Correlation ID: 738ebb7b, Call Stack: null, Custom Event ID: -1, Message: Invalid status on Data Center Default. Setting Data Center status to Non Responsive (On host rack, Error: Cannot access master storage domain). >2014-07-29 00:26:50,949 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-56) [738ebb7b] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2014-07-29 00:26:50,951 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-56) [738ebb7b] hostFromVds::selectedVds - rack, spmStatus returned null! >2014-07-29 00:26:50,952 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-56) [738ebb7b] IrsBroker::Failed::GetStoragePoolInfoVDS >2014-07-29 00:26:50,953 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-56) [738ebb7b] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2014-07-29 00:26:50,988 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-42) [7b0933af] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage >2014-07-29 00:26:51,005 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-42) [7b0933af] START, ResetIrsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, ignoreStopFailed = true), log id: 2b83c9f >2014-07-29 00:26:51,009 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-42) [7b0933af] START, SpmStopVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9), log id: 78bbba55 >2014-07-29 00:26:51,015 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-42) [7b0933af] SpmStopVDSCommand::Stopping SPM on vds rack, pool id 00000002-0002-0002-0002-0000000001d9 >2014-07-29 00:28:50,921 WARN [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-42) [7b0933af] could not stop spm of pool 00000002-0002-0002-0002-0000000001d9 on vds 17f7fe56-5f32-40ef-b098-611b84484e0b - reason: org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcRunTimeException: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException >2014-07-29 00:28:50,924 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-26) Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: VDSNetworkException: org.apache.commons.httpclient.NoHttpResponseException: The server 10.34.62.208 failed to respond >2014-07-29 00:28:50,927 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-26) hostFromVds::selectedVds - rack, spmStatus returned null! >2014-07-29 00:28:50,924 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ListVDSCommand] (DefaultQuartzScheduler_Worker-24) Command ListVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-07-29 00:28:50,931 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-24) Host rack is not responding. It will stay in Connecting state for a grace period of $80 seconds and after that an attempt to fence the host will be issued. >2014-07-29 00:28:50,925 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-42) [7b0933af] FINISH, SpmStopVDSCommand, log id: 78bbba55 >2014-07-29 00:28:50,932 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-24) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:406) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.fetchRunningVms(VdsUpdateRunTimeInfo.java:971) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVmStats(VdsUpdateRunTimeInfo.java:920) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:638) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:235) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) [:1.7.0_55] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_55] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_55] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_55] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_55] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_55] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_55] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_55] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy82.list(Unknown Source) > at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source) [:1.7.0_55] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_55] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_55] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_55] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_55] > >2014-07-29 00:28:50,941 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-42) [7b0933af] FINISH, ResetIrsVDSCommand, log id: 2b83c9f >2014-07-29 00:28:50,967 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-24) Failed to refresh VDS , vds = 17f7fe56-5f32-40ef-b098-611b84484e0b : rack, VDS Network Error, continuing. >java.net.ConnectException: Connection refused >2014-07-29 00:28:50,976 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-42) [7b0933af] Correlation ID: 7b0933af, Job ID: bad0c9c9-9544-4d30-84c9-8154504ac4e8, Call Stack: null, Custom Event ID: -1, Message: Failed to Reconstruct Master Domain for Data Center Default. >2014-07-29 00:28:50,982 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-34) [738ebb7b] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue >2014-07-29 00:28:50,984 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-28) starting processDomainRecovery for domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 >2014-07-29 00:28:51,047 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-28) Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 was reported by all hosts in status UP as problematic. Not moving the domain to NonOperational because it is being reconstructed now. >2014-07-29 00:28:51,067 WARN [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-28) [380ca791] CanDoAction of action ReconstructMasterDomain failed. Reasons:VAR__ACTION__RECONSTRUCT_MASTER,VAR__TYPE__STORAGE__DOMAIN,ACTION_TYPE_FAILED_NO_VDS_IN_POOL >2014-07-29 00:28:51,074 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-28) [380ca791] Removing vds [] from the domain in maintenance cache >2014-07-29 00:28:51,075 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-28) [380ca791] Removing host(s) [] from hosts unseen domain report cache >2014-07-29 00:28:51,076 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-34) [738ebb7b] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue >2014-07-29 00:28:54,232 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-57) START, GetHardwareInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]), log id: 614f08b0 >2014-07-29 00:28:54,286 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-57) FINISH, GetHardwareInfoVDSCommand, log id: 614f08b0 >2014-07-29 00:28:54,317 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-57) [28f420dc] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS >2014-07-29 00:28:54,334 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-57) [7f5d96ca] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-07-29 00:28:54,342 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-57) [1100c7cc] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-07-29 00:28:54,354 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-57) [1100c7cc] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, storageType = NFS, connectionList = [{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 5686beb8 >2014-07-29 00:28:54,407 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-57) [1100c7cc] FINISH, ConnectStorageServerVDSCommand, return: {5083f370-af09-457b-9b73-3440e8b65a07=0}, log id: 5686beb8 >2014-07-29 00:28:54,408 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-57) [1100c7cc] Host rack storage connection was succeeded >2014-07-29 00:28:54,415 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-29) START, ConnectStoragePoolVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, masterVersion = 1), log id: 6071a758 >2014-07-29 00:29:00,931 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-9) Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Inactive to Unknown, reason : no reporting hosts >2014-07-29 00:29:54,806 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-29) FINISH, ConnectStoragePoolVDSCommand, log id: 6071a758 >2014-07-29 00:29:54,820 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-57) [1100c7cc] START, SetMOMPolicyParametersVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 1ef86907 >2014-07-29 00:29:54,914 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-57) [1100c7cc] FINISH, SetMOMPolicyParametersVDSCommand, log id: 1ef86907 >2014-07-29 00:29:54,924 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-57) [1100c7cc] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host rack was set to Up. >2014-07-29 00:29:54,930 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-57) [1100c7cc] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host rack. >2014-07-29 00:29:54,950 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-57) [cd4c3ce] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS >2014-07-29 00:30:01,388 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-27) hostFromVds::selectedVds - rack, spmStatus Free, storage pool Default >2014-07-29 00:30:01,400 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-27) starting spm on vds rack, storage pool Default, prevId -1, LVER -1 >2014-07-29 00:30:01,409 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-27) START, SpmStartVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, prevId=-1, prevLVER=-1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 7a87537 >2014-07-29 00:30:01,459 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-27) spmStart polling started: taskId = 8fcd82c5-f85c-4929-9e85-494fda6c75ed >2014-07-29 00:30:05,480 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-27) spmStart polling ended: taskId = 8fcd82c5-f85c-4929-9e85-494fda6c75ed task status = finished >2014-07-29 00:30:05,536 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-27) spmStart polling ended, spm status: SPM >2014-07-29 00:30:05,539 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-27) START, HSMClearTaskVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, taskId=8fcd82c5-f85c-4929-9e85-494fda6c75ed), log id: be7e3f5 >2014-07-29 00:30:05,584 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-27) FINISH, HSMClearTaskVDSCommand, log id: be7e3f5 >2014-07-29 00:30:05,585 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-27) FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@669cc87, log id: 7a87537 >2014-07-29 00:30:05,589 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-27) Initialize Irs proxy from vds: 10.34.62.208 >2014-07-29 00:30:05,595 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-27) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208). >2014-07-29 00:30:05,609 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-25) START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 7159591f >2014-07-29 00:30:05,634 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-25) -- executeIrsBrokerCommand: Attempting on storage pool 00000002-0002-0002-0002-0000000001d9 >2014-07-29 00:30:05,636 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-25) START, HSMGetAllTasksInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 1d198902 >2014-07-29 00:30:05,641 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-25) FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 1d198902 >2014-07-29 00:30:05,642 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-25) FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 7159591f >2014-07-29 00:30:05,644 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (org.ovirt.thread.pool-8-thread-25) Discovered no tasks on Storage Pool Default >2014-07-29 01:23:54,893 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-67) [2f0280a3] Principal admin::internal synchronized >2014-07-29 01:24:00,917 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-31) Attempting to update VMs/Templates Ovf. >2014-07-29 02:23:54,900 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-81) Principal admin::internal synchronized >2014-07-29 02:24:00,930 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-15) Attempting to update VMs/Templates Ovf. >2014-07-29 03:23:54,907 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-77) Principal admin::internal synchronized >2014-07-29 03:24:00,945 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-8) Attempting to update VMs/Templates Ovf. >2014-07-29 03:35:35,001 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-97) Start deleteAgedOutCommandEntities >2014-07-29 03:35:35,002 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-62) [2e742a17] Start deleteAgedOutAuditLogs >2014-07-29 03:35:35,005 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-62) [2e742a17] Finished deleteAgedOutAuditLogs >2014-07-29 03:35:35,007 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-97) Finished deleteAgedOutCommandEntities >2014-07-29 04:15:31,723 WARN [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-8-thread-5) Executing a command: java.util.concurrent.FutureTask , but note that there are 1 tasks in the queue. >2014-07-29 04:23:54,912 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-54) Principal admin::internal synchronized >2014-07-29 04:24:00,960 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-35) Attempting to update VMs/Templates Ovf. >2014-07-29 05:23:54,919 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-94) Principal admin::internal synchronized >2014-07-29 05:24:00,973 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-27) Attempting to update VMs/Templates Ovf. >2014-07-29 06:23:54,924 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-27) Principal admin::internal synchronized >2014-07-29 06:24:00,988 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-99) Attempting to update VMs/Templates Ovf. >2014-07-29 07:23:54,929 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-33) Principal admin::internal synchronized >2014-07-29 07:24:01,001 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-36) [3e2ffa8] Attempting to update VMs/Templates Ovf. >2014-07-29 07:24:15,032 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-23) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage domain 35 experienced a high latency of 5.7491 seconds from host rack. This may cause performance and functional issues. Please consult your Storage Administrator. >2014-07-29 08:23:54,935 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-35) Principal admin::internal synchronized >2014-07-29 08:24:01,017 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-49) Attempting to update VMs/Templates Ovf. >2014-07-29 09:23:54,940 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-19) Principal admin::internal synchronized >2014-07-29 09:24:01,032 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-84) [7ef2d911] Attempting to update VMs/Templates Ovf. >2014-07-29 09:42:37,236 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-33) domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 in problem. vds: rack >2014-07-29 09:44:02,211 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-41) Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2014-07-29 09:44:02,215 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-41) [5d8b5376] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-07-29 09:44:02,219 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-41) [5d8b5376] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Active to Unknown, reason : null >2014-07-29 09:44:02,233 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-41) [5d8b5376] Correlation ID: 5d8b5376, Call Stack: null, Custom Event ID: -1, Message: Invalid status on Data Center Default. Setting Data Center status to Non Responsive (On host rack, Error: Cannot access master storage domain). >2014-07-29 09:46:02,257 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-41) [5d8b5376] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2014-07-29 09:46:02,260 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-41) [5d8b5376] hostFromVds::selectedVds - rack, spmStatus returned null! >2014-07-29 09:46:02,261 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-41) [5d8b5376] IrsBroker::Failed::GetStoragePoolInfoVDS >2014-07-29 09:46:02,263 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-41) [5d8b5376] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2014-07-29 09:46:02,304 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-37) [1d3e693b] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage >2014-07-29 09:46:02,323 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-37) [1d3e693b] START, ResetIrsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, ignoreStopFailed = true), log id: 6d2622c >2014-07-29 09:46:02,327 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-37) [1d3e693b] START, SpmStopVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9), log id: 5f5b25d8 >2014-07-29 09:46:02,333 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-37) [1d3e693b] SpmStopVDSCommand::Stopping SPM on vds rack, pool id 00000002-0002-0002-0002-0000000001d9 >2014-07-29 09:48:02,228 WARN [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-37) [1d3e693b] could not stop spm of pool 00000002-0002-0002-0002-0000000001d9 on vds 17f7fe56-5f32-40ef-b098-611b84484e0b - reason: org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcRunTimeException: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException >2014-07-29 09:48:02,230 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-37) [1d3e693b] FINISH, SpmStopVDSCommand, log id: 5f5b25d8 >2014-07-29 09:48:02,229 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-83) Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: VDSNetworkException: org.apache.commons.httpclient.NoHttpResponseException: The server 10.34.62.208 failed to respond >2014-07-29 09:48:02,233 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-83) hostFromVds::selectedVds - rack, spmStatus returned null! >2014-07-29 09:48:02,235 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ListVDSCommand] (DefaultQuartzScheduler_Worker-34) Command ListVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-07-29 09:48:02,237 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-34) Host rack is not responding. It will stay in Connecting state for a grace period of $80 seconds and after that an attempt to fence the host will be issued. >2014-07-29 09:48:02,238 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-34) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:406) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.fetchRunningVms(VdsUpdateRunTimeInfo.java:971) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVmStats(VdsUpdateRunTimeInfo.java:920) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:638) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:235) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) [:1.7.0_55] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_55] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_55] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_55] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_55] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_55] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_55] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_55] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy82.list(Unknown Source) > at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source) [:1.7.0_55] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_55] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_55] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_55] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_55] > >2014-07-29 09:48:02,248 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-37) [1d3e693b] FINISH, ResetIrsVDSCommand, log id: 6d2622c >2014-07-29 09:48:02,271 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-34) Failed to refresh VDS , vds = 17f7fe56-5f32-40ef-b098-611b84484e0b : rack, VDS Network Error, continuing. >java.net.ConnectException: Connection refused >2014-07-29 09:48:02,272 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-37) [1d3e693b] Correlation ID: 1d3e693b, Job ID: aefe862a-a37f-4347-ba5e-e324aebf0661, Call Stack: null, Custom Event ID: -1, Message: Failed to Reconstruct Master Domain for Data Center Default. >2014-07-29 09:48:02,279 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-47) [5d8b5376] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue >2014-07-29 09:48:02,281 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-18) starting processDomainRecovery for domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 >2014-07-29 09:48:02,342 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-18) Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 was reported by all hosts in status UP as problematic. Not moving the domain to NonOperational because it is being reconstructed now. >2014-07-29 09:48:02,361 WARN [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-18) [7dffa625] CanDoAction of action ReconstructMasterDomain failed. Reasons:VAR__ACTION__RECONSTRUCT_MASTER,VAR__TYPE__STORAGE__DOMAIN,ACTION_TYPE_FAILED_NO_VDS_IN_POOL >2014-07-29 09:48:02,368 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-18) [7dffa625] Removing vds [] from the domain in maintenance cache >2014-07-29 09:48:02,369 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-18) [7dffa625] Removing host(s) [] from hosts unseen domain report cache >2014-07-29 09:48:02,370 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-47) [5d8b5376] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue >2014-07-29 09:48:05,550 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-95) [75d6be08] START, GetHardwareInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]), log id: 3b7798a4 >2014-07-29 09:48:05,605 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-95) [75d6be08] FINISH, GetHardwareInfoVDSCommand, log id: 3b7798a4 >2014-07-29 09:48:05,637 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-95) [51f62f58] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS >2014-07-29 09:48:05,649 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-95) [46b071d] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-07-29 09:48:05,657 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-95) [c3cc627] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-07-29 09:48:05,668 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-95) [c3cc627] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, storageType = NFS, connectionList = [{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 50722f49 >2014-07-29 09:48:05,730 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-95) [c3cc627] FINISH, ConnectStorageServerVDSCommand, return: {5083f370-af09-457b-9b73-3440e8b65a07=0}, log id: 50722f49 >2014-07-29 09:48:05,731 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-95) [c3cc627] Host rack storage connection was succeeded >2014-07-29 09:48:05,740 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-1) START, ConnectStoragePoolVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, masterVersion = 1), log id: 7ae58a9c >2014-07-29 09:48:12,237 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-17) Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Inactive to Unknown, reason : no reporting hosts >2014-07-29 09:49:14,103 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-1) FINISH, ConnectStoragePoolVDSCommand, log id: 7ae58a9c >2014-07-29 09:49:14,116 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-95) [c3cc627] START, SetMOMPolicyParametersVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 1ba9f8de >2014-07-29 09:49:14,210 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-95) [c3cc627] FINISH, SetMOMPolicyParametersVDSCommand, log id: 1ba9f8de >2014-07-29 09:49:14,218 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-95) [c3cc627] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host rack was set to Up. >2014-07-29 09:49:14,225 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-95) [c3cc627] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host rack. >2014-07-29 09:49:14,242 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-95) [72971053] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS >2014-07-29 09:49:22,356 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-64) hostFromVds::selectedVds - rack, spmStatus Free, storage pool Default >2014-07-29 09:49:22,361 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-64) starting spm on vds rack, storage pool Default, prevId -1, LVER -1 >2014-07-29 09:49:22,364 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-64) START, SpmStartVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, prevId=-1, prevLVER=-1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 17059a88 >2014-07-29 09:49:22,412 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-64) spmStart polling started: taskId = 2621acf2-1ca0-453a-b925-4b9ff956e6e9 >2014-07-29 09:49:23,417 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-64) spmStart polling ended: taskId = 2621acf2-1ca0-453a-b925-4b9ff956e6e9 task status = finished >2014-07-29 09:49:23,473 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-64) spmStart polling ended, spm status: SPM >2014-07-29 09:49:23,476 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-64) START, HSMClearTaskVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, taskId=2621acf2-1ca0-453a-b925-4b9ff956e6e9), log id: 1800b4f4 >2014-07-29 09:49:23,521 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-64) FINISH, HSMClearTaskVDSCommand, log id: 1800b4f4 >2014-07-29 09:49:23,523 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-64) FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@57c761b7, log id: 17059a88 >2014-07-29 09:49:23,527 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-64) Initialize Irs proxy from vds: 10.34.62.208 >2014-07-29 09:49:23,534 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-64) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208). >2014-07-29 09:49:23,536 WARN [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-8-thread-24) Executing a command: java.util.concurrent.FutureTask , but note that there are 1 tasks in the queue. >2014-07-29 09:49:23,547 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-24) START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 670d4aca >2014-07-29 09:49:23,569 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-24) -- executeIrsBrokerCommand: Attempting on storage pool 00000002-0002-0002-0002-0000000001d9 >2014-07-29 09:49:23,571 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-24) START, HSMGetAllTasksInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 532d9fb4 >2014-07-29 09:49:23,576 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-24) FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 532d9fb4 >2014-07-29 09:49:23,577 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-24) FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 670d4aca >2014-07-29 09:49:23,578 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (org.ovirt.thread.pool-8-thread-24) Discovered no tasks on Storage Pool Default >2014-07-29 10:23:54,947 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-99) Principal admin::internal synchronized >2014-07-29 10:24:01,047 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-91) Attempting to update VMs/Templates Ovf. >2014-07-29 10:28:11,818 INFO [org.ovirt.engine.core.bll.LoginAdminUserCommand] (ajp--127.0.0.1-8702-5) Running command: LoginAdminUserCommand internal: false. >2014-07-29 10:28:11,824 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-5) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-07-29 10:28:12,477 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp--127.0.0.1-8702-3) Running command: LoginUserCommand internal: false. >2014-07-29 10:49:12,454 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp--127.0.0.1-8702-4) Running command: LoginUserCommand internal: false. >2014-07-29 10:49:12,459 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-4) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-07-29 10:49:13,349 INFO [org.ovirt.engine.core.bll.LogoutBySessionCommand] (ajp--127.0.0.1-8702-4) [3694f201] Running command: LogoutBySessionCommand internal: false. >2014-07-29 10:49:13,356 INFO [org.ovirt.engine.core.bll.LogoutUserCommand] (ajp--127.0.0.1-8702-4) [6a24efe7] Running command: LogoutUserCommand internal: false. >2014-07-29 10:49:13,362 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-4) [6a24efe7] Correlation ID: 6a24efe7, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-07-29 10:49:13,769 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp--127.0.0.1-8702-1) Running command: LoginUserCommand internal: false. >2014-07-29 10:49:13,783 INFO [org.ovirt.engine.core.bll.LogoutBySessionCommand] (ajp--127.0.0.1-8702-1) [3ebcd9a] Running command: LogoutBySessionCommand internal: false. >2014-07-29 10:49:13,787 INFO [org.ovirt.engine.core.bll.LogoutUserCommand] (ajp--127.0.0.1-8702-1) [579bdb1b] Running command: LogoutUserCommand internal: false. >2014-07-29 10:49:13,792 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-1) [579bdb1b] Correlation ID: 579bdb1b, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-07-29 10:49:26,513 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp--127.0.0.1-8702-3) Running command: LoginUserCommand internal: false. >2014-07-29 10:49:26,517 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-3) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-07-29 10:49:26,547 INFO [org.ovirt.engine.core.bll.LogoutBySessionCommand] (ajp--127.0.0.1-8702-3) [1dbb305f] Running command: LogoutBySessionCommand internal: false. >2014-07-29 10:49:26,552 INFO [org.ovirt.engine.core.bll.LogoutUserCommand] (ajp--127.0.0.1-8702-3) [180692ea] Running command: LogoutUserCommand internal: false. >2014-07-29 10:49:26,563 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-3) [180692ea] Correlation ID: 180692ea, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-07-29 10:49:46,382 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp--127.0.0.1-8702-7) Running command: LoginUserCommand internal: false. >2014-07-29 10:49:46,386 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-7) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-07-29 10:49:48,310 INFO [org.ovirt.engine.core.bll.LogoutBySessionCommand] (ajp--127.0.0.1-8702-7) [5bcc8676] Running command: LogoutBySessionCommand internal: false. >2014-07-29 10:49:48,312 INFO [org.ovirt.engine.core.bll.LogoutUserCommand] (ajp--127.0.0.1-8702-7) [4fc67a4c] Running command: LogoutUserCommand internal: false. >2014-07-29 10:49:48,316 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-7) [4fc67a4c] Correlation ID: 4fc67a4c, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-07-29 10:50:44,485 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp--127.0.0.1-8702-2) Running command: LoginUserCommand internal: false. >2014-07-29 10:50:44,489 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-2) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-07-29 10:50:45,275 INFO [org.ovirt.engine.core.bll.LogoutBySessionCommand] (ajp--127.0.0.1-8702-2) [3a9f0743] Running command: LogoutBySessionCommand internal: false. >2014-07-29 10:50:45,278 INFO [org.ovirt.engine.core.bll.LogoutUserCommand] (ajp--127.0.0.1-8702-2) [65464a80] Running command: LogoutUserCommand internal: false. >2014-07-29 10:50:45,282 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-2) [65464a80] Correlation ID: 65464a80, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-07-29 10:50:45,625 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp--127.0.0.1-8702-5) Running command: LoginUserCommand internal: false. >2014-07-29 10:50:45,665 INFO [org.ovirt.engine.core.bll.LogoutBySessionCommand] (ajp--127.0.0.1-8702-5) [44d6b145] Running command: LogoutBySessionCommand internal: false. >2014-07-29 10:50:45,667 INFO [org.ovirt.engine.core.bll.LogoutUserCommand] (ajp--127.0.0.1-8702-5) [48b3bc09] Running command: LogoutUserCommand internal: false. >2014-07-29 10:50:45,670 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-5) [48b3bc09] Correlation ID: 48b3bc09, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-07-29 10:52:27,795 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp--127.0.0.1-8702-3) Running command: LoginUserCommand internal: false. >2014-07-29 10:52:27,801 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-3) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-07-29 10:52:27,808 INFO [org.ovirt.engine.core.bll.LogoutBySessionCommand] (ajp--127.0.0.1-8702-3) [38f821c0] Running command: LogoutBySessionCommand internal: false. >2014-07-29 10:52:27,813 INFO [org.ovirt.engine.core.bll.LogoutUserCommand] (ajp--127.0.0.1-8702-3) [5f8314d9] Running command: LogoutUserCommand internal: false. >2014-07-29 10:52:27,818 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-3) [5f8314d9] Correlation ID: 5f8314d9, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-07-29 10:52:36,559 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp--127.0.0.1-8702-1) Running command: LoginUserCommand internal: false. >2014-07-29 10:52:36,563 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-1) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-07-29 10:52:37,503 INFO [org.ovirt.engine.core.bll.LogoutBySessionCommand] (ajp--127.0.0.1-8702-1) [30b339af] Running command: LogoutBySessionCommand internal: false. >2014-07-29 10:52:37,505 INFO [org.ovirt.engine.core.bll.LogoutUserCommand] (ajp--127.0.0.1-8702-1) [3ded41b5] Running command: LogoutUserCommand internal: false. >2014-07-29 10:52:37,509 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-1) [3ded41b5] Correlation ID: 3ded41b5, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-07-29 10:53:10,058 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp--127.0.0.1-8702-4) Running command: LoginUserCommand internal: false. >2014-07-29 10:53:10,061 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-4) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-07-29 10:54:40,251 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp--127.0.0.1-8702-2) Running command: LoginUserCommand internal: false. >2014-07-29 10:54:40,260 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-2) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-07-29 10:54:40,267 INFO [org.ovirt.engine.core.bll.LogoutBySessionCommand] (ajp--127.0.0.1-8702-2) [31145d7b] Running command: LogoutBySessionCommand internal: false. >2014-07-29 10:54:40,270 INFO [org.ovirt.engine.core.bll.LogoutUserCommand] (ajp--127.0.0.1-8702-2) [6cbf2fba] Running command: LogoutUserCommand internal: false. >2014-07-29 10:54:40,274 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-2) [6cbf2fba] Correlation ID: 6cbf2fba, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-07-29 10:55:40,177 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp--127.0.0.1-8702-11) Running command: LoginUserCommand internal: false. >2014-07-29 10:55:40,181 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-11) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-07-29 10:55:40,187 INFO [org.ovirt.engine.core.bll.LogoutBySessionCommand] (ajp--127.0.0.1-8702-11) [6cb66736] Running command: LogoutBySessionCommand internal: false. >2014-07-29 10:55:40,196 INFO [org.ovirt.engine.core.bll.LogoutUserCommand] (ajp--127.0.0.1-8702-11) [34524ed8] Running command: LogoutUserCommand internal: false. >2014-07-29 10:55:40,203 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-11) [34524ed8] Correlation ID: 34524ed8, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-07-29 11:23:54,956 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-98) [5aa74089] Principal admin::internal synchronized >2014-07-29 11:24:01,062 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-74) Attempting to update VMs/Templates Ovf. >2014-07-29 12:23:54,962 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-43) Principal admin::internal synchronized >2014-07-29 12:24:01,076 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-13) [4dca0641] Attempting to update VMs/Templates Ovf. >2014-07-29 12:58:21,049 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-22) domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 in problem. vds: rack >2014-07-29 12:58:36,139 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-32) Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 recovered from problem. vds: rack >2014-07-29 12:58:36,140 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-32) Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 has recovered from problem. No active host in the DC is reporting it as problematic, so clearing the domain recovery timer. >2014-07-29 12:59:21,621 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-83) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage domain 35 experienced a high latency of 5.60819 seconds from host rack. This may cause performance and functional issues. Please consult your Storage Administrator. >2014-07-29 13:23:54,966 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-15) Principal admin::internal synchronized >2014-07-29 13:24:01,088 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-59) Attempting to update VMs/Templates Ovf. >2014-07-29 14:00:21,061 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-48) domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 in problem. vds: rack >2014-07-29 14:00:36,191 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-28) Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 recovered from problem. vds: rack >2014-07-29 14:00:36,193 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-28) Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 has recovered from problem. No active host in the DC is reporting it as problematic, so clearing the domain recovery timer. >2014-07-29 14:01:21,593 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-55) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage domain 35 experienced a high latency of 7.1651 seconds from host rack. This may cause performance and functional issues. Please consult your Storage Administrator. >2014-07-29 14:23:54,973 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-59) Principal admin::internal synchronized >2014-07-29 14:24:01,102 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-29) [327b509d] Attempting to update VMs/Templates Ovf. >2014-07-29 14:25:48,963 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-6) domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 in problem. vds: rack >2014-07-29 14:26:04,088 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-39) Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 recovered from problem. vds: rack >2014-07-29 14:26:04,089 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-39) Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 has recovered from problem. No active host in the DC is reporting it as problematic, so clearing the domain recovery timer. >2014-07-29 15:23:54,980 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-48) Principal admin::internal synchronized >2014-07-29 15:24:01,117 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-29) [327b509d] Attempting to update VMs/Templates Ovf. >2014-07-29 15:44:11,493 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-1) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage domain 35 experienced a high latency of 5.1973 seconds from host rack. This may cause performance and functional issues. Please consult your Storage Administrator. >2014-07-29 16:05:06,478 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-45) domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 in problem. vds: rack >2014-07-29 16:05:21,601 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-47) Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 recovered from problem. vds: rack >2014-07-29 16:05:21,603 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-47) Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 has recovered from problem. No active host in the DC is reporting it as problematic, so clearing the domain recovery timer. >2014-07-29 16:23:54,986 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-83) Principal admin::internal synchronized >2014-07-29 16:24:01,148 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-62) [2e742a17] Attempting to update VMs/Templates Ovf. >2014-07-29 16:47:17,745 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-22) Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-07-29 16:47:17,749 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-22) [7e0971f4] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-07-29 16:47:17,756 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-22) [7e0971f4] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Active to Unknown, reason : null >2014-07-29 16:47:17,772 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ListVDSCommand] (DefaultQuartzScheduler_Worker-82) [3abdaaa6] Command ListVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-07-29 16:47:17,775 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-82) [3abdaaa6] Host rack is not responding. It will stay in Connecting state for a grace period of $80 seconds and after that an attempt to fence the host will be issued. >2014-07-29 16:47:17,784 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-22) [7e0971f4] Correlation ID: 7e0971f4, Call Stack: null, Custom Event ID: -1, Message: Invalid status on Data Center Default. Setting Data Center status to Non Responsive (On host rack, Error: Network error during communication with the Host.). >2014-07-29 16:47:17,778 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-82) [3abdaaa6] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:406) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.fetchRunningVms(VdsUpdateRunTimeInfo.java:971) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVmStats(VdsUpdateRunTimeInfo.java:920) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:638) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:235) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) [:1.7.0_55] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_55] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_55] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_55] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_55] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_55] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_55] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_55] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy82.list(Unknown Source) > at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source) [:1.7.0_55] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_55] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_55] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_55] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_55] > >2014-07-29 16:47:17,809 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-22) [7e0971f4] IrsBroker::Failed::GetStoragePoolInfoVDS due to: ConnectException: Connection refused >2014-07-29 16:47:17,842 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-82) [3abdaaa6] Failed to refresh VDS , vds = 17f7fe56-5f32-40ef-b098-611b84484e0b : rack, VDS Network Error, continuing. >java.net.ConnectException: Connection refused >2014-07-29 16:47:20,945 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-95) [72971053] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@5c539198 >2014-07-29 16:47:20,946 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-95) [72971053] HostName = rack >2014-07-29 16:47:20,947 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-95) [72971053] Command GetCapabilitiesVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]) execution failed. Exception: VDSRecoveringException: Recovering from crash or Initializing >2014-07-29 16:47:20,960 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-95) [72971053] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSRecoveringException: Recovering from crash or Initializing > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:16) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:566) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:235) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Host rack is initializing. Message: Recovering from crash or Initializing >2014-07-29 16:47:20,967 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-95) [72971053] Failed to refresh VDS , vds = 17f7fe56-5f32-40ef-b098-611b84484e0b : rack, error = Recovering from crash or Initializing, continuing. >2014-07-29 16:47:24,234 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-51) START, GetHardwareInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]), log id: 64603823 >2014-07-29 16:47:24,312 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-51) FINISH, GetHardwareInfoVDSCommand, log id: 64603823 >2014-07-29 16:47:24,346 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-51) [e91c5b2] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS >2014-07-29 16:47:24,355 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-51) [e91c5b2] Cancelling the recovery from crash timer for VDS 17f7fe56-5f32-40ef-b098-611b84484e0b because vds started initializing >2014-07-29 16:47:24,357 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-51) [1134a0fc] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-07-29 16:47:24,369 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-51) [7df1d9db] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-07-29 16:47:24,380 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-51) [7df1d9db] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, storageType = NFS, connectionList = [{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 5f7ebc9b >2014-07-29 16:47:24,436 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-51) [7df1d9db] FINISH, ConnectStorageServerVDSCommand, return: {5083f370-af09-457b-9b73-3440e8b65a07=0}, log id: 5f7ebc9b >2014-07-29 16:47:24,437 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-51) [7df1d9db] Host rack storage connection was succeeded >2014-07-29 16:47:24,444 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-21) START, ConnectStoragePoolVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, masterVersion = 1), log id: 2600cee >2014-07-29 16:47:27,944 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-28) [8ef1638] Command org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand return value > >TaskStatusListReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=654, mMessage=Not SPM: ()]] > >2014-07-29 16:47:27,946 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-28) [8ef1638] HostName = rack >2014-07-29 16:47:27,947 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-28) [8ef1638] Command HSMGetAllTasksStatusesVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b) execution failed. Exception: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM: () >2014-07-29 16:47:39,313 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-21) Command ConnectStoragePoolVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, masterVersion = 1) execution failed. Exception: VDSNetworkException: org.apache.commons.httpclient.NoHttpResponseException: The server 10.34.62.208 failed to respond >2014-07-29 16:47:39,315 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-21) FINISH, ConnectStoragePoolVDSCommand, log id: 2600cee >2014-07-29 16:47:39,316 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-8-thread-21) Could not connect host rack to pool Default, as the master domain is in inactive/unknown status - not failing the operation >2014-07-29 16:47:39,320 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetStatsVDSCommand] (org.ovirt.thread.pool-8-thread-21) Command GetStatsVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-07-29 16:47:39,322 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-8-thread-21) Could not get Host statistics for Host rack, Error is org.ovirt.engine.core.common.errors.VdcBLLException: VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused (Failed with error VDS_NETWORK_ERROR and code 5022): org.ovirt.engine.core.common.errors.VdcBLLException: VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused (Failed with error VDS_NETWORK_ERROR and code 5022) > at org.ovirt.engine.core.bll.VdsHandler.handleVdsResult(VdsHandler.java:116) [bll.jar:] > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) [bll.jar:] > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1971) [bll.jar:] > at org.ovirt.engine.core.bll.storage.StorageHandlingCommandBase.runVdsCommand(StorageHandlingCommandBase.java:627) [bll.jar:] > at org.ovirt.engine.core.bll.InitVdsOnUpCommand.proceedVdsStats(InitVdsOnUpCommand.java:306) [bll.jar:] > at org.ovirt.engine.core.bll.InitVdsOnUpCommand.runConnectHostToPoolEvent(InitVdsOnUpCommand.java:276) [bll.jar:] > at org.ovirt.engine.core.bll.InitVdsOnUpCommand.access$000(InitVdsOnUpCommand.java:79) [bll.jar:] > at org.ovirt.engine.core.bll.InitVdsOnUpCommand$1.call(InitVdsOnUpCommand.java:237) [bll.jar:] > at org.ovirt.engine.core.bll.InitVdsOnUpCommand$1.call(InitVdsOnUpCommand.java:234) [bll.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_55] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_55] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_55] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_55] > >2014-07-29 16:47:39,336 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-21) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host rack reports about one of the Active Storage Domains as Problematic. >2014-07-29 16:47:39,358 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-51) [654b0a57] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS >2014-07-29 16:47:39,362 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-51) [654b0a57] START, SetVdsStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 3a0d438e >2014-07-29 16:47:39,364 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-51) [654b0a57] VDS rack is spm and moved from up calling resetIrs. >2014-07-29 16:47:39,365 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (DefaultQuartzScheduler_Worker-51) [654b0a57] START, ResetIrsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, ignoreStopFailed = false), log id: 2f5ddc2e >2014-07-29 16:47:39,368 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (DefaultQuartzScheduler_Worker-51) [654b0a57] START, SpmStopVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9), log id: 59f7ce8a >2014-07-29 16:47:39,373 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-51) [654b0a57] Command HSMGetAllTasksStatusesVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-07-29 16:47:39,375 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (DefaultQuartzScheduler_Worker-51) [654b0a57] SpmStopVDSCommand::Stopping SPM on vds rack, pool id 00000002-0002-0002-0002-0000000001d9 >2014-07-29 16:47:39,382 WARN [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (DefaultQuartzScheduler_Worker-51) [654b0a57] could not stop spm of pool 00000002-0002-0002-0002-0000000001d9 on vds 17f7fe56-5f32-40ef-b098-611b84484e0b - reason: org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcRunTimeException: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException >2014-07-29 16:47:39,384 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (DefaultQuartzScheduler_Worker-51) [654b0a57] FINISH, SpmStopVDSCommand, log id: 59f7ce8a >2014-07-29 16:47:39,385 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (DefaultQuartzScheduler_Worker-51) [654b0a57] FINISH, ResetIrsVDSCommand, log id: 2f5ddc2e >2014-07-29 16:47:39,391 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-51) [654b0a57] FINISH, SetVdsStatusVDSCommand, log id: 3a0d438e >2014-07-29 16:47:39,415 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-51) [654b0a57] Correlation ID: 654b0a57, Job ID: 769ebd00-e82d-4a7c-8271-5b8eb1c0e432, Call Stack: null, Custom Event ID: -1, Message: Host rack cannot access the Storage Domain(s) Non interactive user attached to the Data Center Default. Setting Host state to Non-Operational. >2014-07-29 16:47:39,428 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-51) [654b0a57] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host rack. >2014-07-29 16:47:39,435 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-51) [654b0a57] Correlation ID: 1134a0fc, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host rack to Storage Pool Default >2014-07-29 16:47:39,452 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-51) [4760e42] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS >2014-07-29 16:47:39,454 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-51) [4760e42] Host 17f7fe56-5f32-40ef-b098-611b84484e0b : rack is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-07-29 16:47:39,490 INFO [org.ovirt.engine.core.bll.InternalMigrateVmCommand] (org.ovirt.thread.pool-8-thread-48) [601efbc8] Lock Acquired to object EngineLock [exclusiveLocks= key: 78722009-2727-4715-91bf-dbbb33ce111e value: VM >, sharedLocks= ] >2014-07-29 16:47:39,554 WARN [org.ovirt.engine.core.bll.InternalMigrateVmCommand] (org.ovirt.thread.pool-8-thread-48) [601efbc8] CanDoAction of action InternalMigrateVm failed. Reasons:VAR__ACTION__MIGRATE,VAR__TYPE__VM,SCHEDULING_NO_HOSTS >2014-07-29 16:47:39,555 INFO [org.ovirt.engine.core.bll.InternalMigrateVmCommand] (org.ovirt.thread.pool-8-thread-48) [601efbc8] Lock freed to object EngineLock [exclusiveLocks= key: 78722009-2727-4715-91bf-dbbb33ce111e value: VM >, sharedLocks= ] >2014-07-29 16:47:39,563 ERROR [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (org.ovirt.thread.pool-8-thread-48) [601efbc8] ResourceManager::vdsMaintenance - Failed migrating desktop testtesttest >2014-07-29 16:47:42,465 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetStatsVDSCommand] (DefaultQuartzScheduler_Worker-24) Command GetStatsVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-07-29 16:47:42,468 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-24) Host rack is not responding. It will stay in Connecting state for a grace period of $80 seconds and after that an attempt to fence the host will be issued. >2014-07-29 16:47:42,469 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-24) vds::refreshVdsStats Failed getVdsStats, vds = 17f7fe56-5f32-40ef-b098-611b84484e0b : rack, error = VDSNetworkException: java.net.ConnectException: Connection refused >2014-07-29 16:47:42,471 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-24) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsStats(VdsUpdateRunTimeInfo.java:669) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:623) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:235) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) [:1.7.0_55] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_55] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_55] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_55] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_55] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_55] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_55] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_55] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy82.getVdsStats(Unknown Source) > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_55] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_55] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_55] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_55] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_55] > >2014-07-29 16:47:42,501 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-24) Failed to refresh VDS , vds = 17f7fe56-5f32-40ef-b098-611b84484e0b : rack, VDS Network Error, continuing. >java.net.ConnectException: Connection refused >2014-07-29 16:47:45,510 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-20) Command GetCapabilitiesVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-07-29 16:47:45,512 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-20) Host rack is not responding. It will stay in Connecting state for a grace period of $80 seconds and after that an attempt to fence the host will be issued. >2014-07-29 16:47:45,513 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-20) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:566) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:235) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) [:1.7.0_55] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_55] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_55] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_55] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_55] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_55] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_55] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_55] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy82.getVdsCapabilities(Unknown Source) > at sun.reflect.GeneratedMethodAccessor429.invoke(Unknown Source) [:1.7.0_55] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_55] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_55] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_55] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_55] > >2014-07-29 16:47:45,543 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-20) Failed to refresh VDS , vds = 17f7fe56-5f32-40ef-b098-611b84484e0b : rack, VDS Network Error, continuing. >java.net.ConnectException: Connection refused >2014-07-29 16:47:48,653 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-52) Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@2e309e7f >2014-07-29 16:47:48,654 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-52) HostName = rack >2014-07-29 16:47:48,655 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-52) Command GetCapabilitiesVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]) execution failed. Exception: VDSRecoveringException: Recovering from crash or Initializing >2014-07-29 16:47:48,665 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-52) Failed to refresh VDS , vds = 17f7fe56-5f32-40ef-b098-611b84484e0b : rack, error = Recovering from crash or Initializing, continuing. >2014-07-29 16:47:52,097 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-30) START, GetHardwareInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]), log id: 3c8a726d >2014-07-29 16:47:52,147 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-30) FINISH, GetHardwareInfoVDSCommand, log id: 3c8a726d >2014-07-29 16:47:52,171 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-30) [2160db07] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS >2014-07-29 16:47:52,221 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-30) [2160db07] Cancelling the recovery from crash timer for VDS 17f7fe56-5f32-40ef-b098-611b84484e0b because vds started initializing >2014-07-29 16:47:52,224 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-30) [78371c83] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-07-29 16:47:52,231 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-30) [1ba0dab7] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-07-29 16:47:52,238 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-30) [1ba0dab7] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, storageType = NFS, connectionList = [{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 38b36d36 >2014-07-29 16:47:52,290 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-30) [1ba0dab7] FINISH, ConnectStorageServerVDSCommand, return: {5083f370-af09-457b-9b73-3440e8b65a07=0}, log id: 38b36d36 >2014-07-29 16:47:52,292 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-30) [1ba0dab7] Host rack storage connection was succeeded >2014-07-29 16:47:52,299 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-33) START, ConnectStoragePoolVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, masterVersion = 1), log id: 7a80b6fd >2014-07-29 16:48:52,654 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-33) FINISH, ConnectStoragePoolVDSCommand, log id: 7a80b6fd >2014-07-29 16:48:52,922 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-30) [1ba0dab7] START, SetMOMPolicyParametersVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 5756c566 >2014-07-29 16:48:53,018 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-30) [1ba0dab7] FINISH, SetMOMPolicyParametersVDSCommand, log id: 5756c566 >2014-07-29 16:48:53,025 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-30) [1ba0dab7] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host rack was set to Up. >2014-07-29 16:48:53,031 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-30) [1ba0dab7] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host rack. >2014-07-29 16:48:53,050 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-30) [dd7f8b1] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS >2014-07-29 16:48:58,108 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-17) hostFromVds::selectedVds - rack, spmStatus Free, storage pool Default >2014-07-29 16:48:58,113 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-17) starting spm on vds rack, storage pool Default, prevId -1, LVER -1 >2014-07-29 16:48:58,116 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-17) START, SpmStartVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, prevId=-1, prevLVER=-1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 54a9cff5 >2014-07-29 16:48:58,165 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-17) spmStart polling started: taskId = 293fb8a0-656c-4123-aacc-626905de7937 >2014-07-29 16:48:59,170 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-17) spmStart polling ended: taskId = 293fb8a0-656c-4123-aacc-626905de7937 task status = finished >2014-07-29 16:48:59,228 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-17) spmStart polling ended, spm status: SPM >2014-07-29 16:48:59,230 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-17) START, HSMClearTaskVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, taskId=293fb8a0-656c-4123-aacc-626905de7937), log id: 2fa7097b >2014-07-29 16:48:59,276 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-17) FINISH, HSMClearTaskVDSCommand, log id: 2fa7097b >2014-07-29 16:48:59,276 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-17) FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@4dab6934, log id: 54a9cff5 >2014-07-29 16:48:59,280 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-17) Initialize Irs proxy from vds: 10.34.62.208 >2014-07-29 16:48:59,286 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-17) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208). >2014-07-29 16:48:59,298 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-50) START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: f8caf47 >2014-07-29 16:48:59,321 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-50) -- executeIrsBrokerCommand: Attempting on storage pool 00000002-0002-0002-0002-0000000001d9 >2014-07-29 16:48:59,323 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-50) START, HSMGetAllTasksInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 70aad3f8 >2014-07-29 16:48:59,329 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-50) FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 70aad3f8 >2014-07-29 16:48:59,330 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-50) FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: f8caf47 >2014-07-29 16:48:59,331 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (org.ovirt.thread.pool-8-thread-50) Discovered no tasks on Storage Pool Default >2014-07-29 17:03:10,268 INFO [org.ovirt.engine.core.bll.MaintenanceNumberOfVdssCommand] (ajp--127.0.0.1-8702-5) [3d8b71da] Lock Acquired to object EngineLock [exclusiveLocks= , sharedLocks= key: 00000002-0002-0002-0002-0000000001d9 value: POOL >] >2014-07-29 17:03:10,334 WARN [org.ovirt.engine.core.bll.MaintenanceNumberOfVdssCommand] (ajp--127.0.0.1-8702-5) [3d8b71da] CanDoAction of action MaintenanceNumberOfVdss failed. Reasons:VAR__TYPE__HOST,VAR__ACTION__MAINTENANCE,CANNOT_MAINTENANCE_VDS_RUN_VMS_NO_OTHER_RUNNING_VDS,$ClustersList Default,$HostsList rack >2014-07-29 17:03:10,336 INFO [org.ovirt.engine.core.bll.MaintenanceNumberOfVdssCommand] (ajp--127.0.0.1-8702-5) [3d8b71da] Lock freed to object EngineLock [exclusiveLocks= , sharedLocks= key: 00000002-0002-0002-0002-0000000001d9 value: POOL >] >2014-07-29 17:03:24,889 INFO [org.ovirt.engine.core.bll.RunVmCommand] (ajp--127.0.0.1-8702-5) [95994fa] Lock Acquired to object EngineLock [exclusiveLocks= key: 78722009-2727-4715-91bf-dbbb33ce111e value: VM >, sharedLocks= ] >2014-07-29 17:03:24,986 INFO [org.ovirt.engine.core.bll.RunVmCommand] (org.ovirt.thread.pool-8-thread-34) [95994fa] Running command: RunVmCommand internal: false. Entities affected : ID: 78722009-2727-4715-91bf-dbbb33ce111e Type: VM >2014-07-29 17:03:24,993 INFO [org.ovirt.engine.core.vdsbroker.ResumeVDSCommand] (org.ovirt.thread.pool-8-thread-34) [95994fa] START, ResumeVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vmId=78722009-2727-4715-91bf-dbbb33ce111e), log id: 7be2f515 >2014-07-29 17:03:24,997 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ResumeBrokerVDSCommand] (org.ovirt.thread.pool-8-thread-34) [95994fa] START, ResumeBrokerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vmId=78722009-2727-4715-91bf-dbbb33ce111e), log id: 1c7bdac5 >2014-07-29 17:03:25,046 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ResumeBrokerVDSCommand] (org.ovirt.thread.pool-8-thread-34) [95994fa] FINISH, ResumeBrokerVDSCommand, log id: 1c7bdac5 >2014-07-29 17:03:25,047 INFO [org.ovirt.engine.core.vdsbroker.ResumeVDSCommand] (org.ovirt.thread.pool-8-thread-34) [95994fa] FINISH, ResumeVDSCommand, return: PoweringUp, log id: 7be2f515 >2014-07-29 17:03:25,048 INFO [org.ovirt.engine.core.bll.RunVmCommand] (org.ovirt.thread.pool-8-thread-34) [95994fa] Lock freed to object EngineLock [exclusiveLocks= key: 78722009-2727-4715-91bf-dbbb33ce111e value: VM >, sharedLocks= ] >2014-07-29 17:03:25,063 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-34) [95994fa] Correlation ID: 95994fa, Job ID: ab7eff79-8541-41d9-99c3-000b056e06c8, Call Stack: null, Custom Event ID: -1, Message: VM testtesttest was resumed by admin (Host: rack). >2014-07-29 17:03:27,533 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-25) VM testtesttest 78722009-2727-4715-91bf-dbbb33ce111e moved from Paused --> Up >2014-07-29 17:03:27,547 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-25) Correlation ID: 95994fa, Job ID: ab7eff79-8541-41d9-99c3-000b056e06c8, Call Stack: null, Custom Event ID: -1, Message: VM testtesttest started on Host rack >2014-07-29 17:05:40,708 INFO [org.ovirt.engine.core.bll.ShutdownVmCommand] (org.ovirt.thread.pool-8-thread-50) [59cf20f4] Running command: ShutdownVmCommand internal: false. Entities affected : ID: 78722009-2727-4715-91bf-dbbb33ce111e Type: VM >2014-07-29 17:05:40,712 INFO [org.ovirt.engine.core.bll.ShutdownVmCommand] (org.ovirt.thread.pool-8-thread-50) [59cf20f4] Entered (VM testtesttest). >2014-07-29 17:05:40,713 INFO [org.ovirt.engine.core.bll.ShutdownVmCommand] (org.ovirt.thread.pool-8-thread-50) [59cf20f4] Sending shutdown command for VM testtesttest. >2014-07-29 17:05:40,716 INFO [org.ovirt.engine.core.vdsbroker.DestroyVmVDSCommand] (org.ovirt.thread.pool-8-thread-50) [59cf20f4] START, DestroyVmVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vmId=78722009-2727-4715-91bf-dbbb33ce111e, force=false, secondsToWait=30, gracefully=true, reason=), log id: 5e9a73e1 >2014-07-29 17:05:40,722 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (org.ovirt.thread.pool-8-thread-50) [59cf20f4] START, DestroyVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vmId=78722009-2727-4715-91bf-dbbb33ce111e, force=false, secondsToWait=30, gracefully=true, reason=), log id: 9d52f7b >2014-07-29 17:05:40,728 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (org.ovirt.thread.pool-8-thread-50) [59cf20f4] FINISH, DestroyVDSCommand, log id: 9d52f7b >2014-07-29 17:05:40,735 INFO [org.ovirt.engine.core.vdsbroker.DestroyVmVDSCommand] (org.ovirt.thread.pool-8-thread-50) [59cf20f4] FINISH, DestroyVmVDSCommand, return: PoweringDown, log id: 5e9a73e1 >2014-07-29 17:05:40,743 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-50) [59cf20f4] Correlation ID: 59cf20f4, Job ID: f2a8e708-7284-4867-9a62-9941ca4445e6, Call Stack: null, Custom Event ID: -1, Message: VM shutdown initiated by admin on VM testtesttest (Host: rack) (Reason: Not Specified). >2014-07-29 17:05:58,900 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (DefaultQuartzScheduler_Worker-95) [72971053] START, DestroyVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vmId=78722009-2727-4715-91bf-dbbb33ce111e, force=false, secondsToWait=0, gracefully=false, reason=), log id: 6cc96aca >2014-07-29 17:05:58,947 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (DefaultQuartzScheduler_Worker-95) [72971053] FINISH, DestroyVDSCommand, log id: 6cc96aca >2014-07-29 17:05:59,014 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-95) [72971053] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: VM testtesttest is down. Exit message: User shut down from within the guest >2014-07-29 17:05:59,017 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-95) [72971053] VM testtesttest (78722009-2727-4715-91bf-dbbb33ce111e) is running in db and not running in VDS rack >2014-07-29 17:05:59,018 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVdsCommand] (DefaultQuartzScheduler_Worker-95) [72971053] START, FullListVdsCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b], vmIds=[78722009-2727-4715-91bf-dbbb33ce111e]), log id: 447b6aa4 >2014-07-29 17:05:59,023 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVdsCommand] (DefaultQuartzScheduler_Worker-95) [72971053] FINISH, FullListVdsCommand, return: [], log id: 447b6aa4 >2014-07-29 17:05:59,039 INFO [org.ovirt.engine.core.bll.ProcessDownVmCommand] (org.ovirt.thread.pool-8-thread-15) [77cf3175] Running command: ProcessDownVmCommand internal: true. >2014-07-29 17:07:27,287 INFO [org.ovirt.engine.core.bll.MaintenanceNumberOfVdssCommand] (ajp--127.0.0.1-8702-1) [9a2e425] Lock Acquired to object EngineLock [exclusiveLocks= , sharedLocks= key: 00000002-0002-0002-0002-0000000001d9 value: POOL >] >2014-07-29 17:07:27,324 INFO [org.ovirt.engine.core.bll.MaintenanceNumberOfVdssCommand] (org.ovirt.thread.pool-8-thread-48) [9a2e425] Running command: MaintenanceNumberOfVdssCommand internal: false. Entities affected : ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS >2014-07-29 17:07:27,335 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-8-thread-48) [9a2e425] START, SetVdsStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, status=PreparingForMaintenance, nonOperationalReason=NONE, stopSpmFailureLogged=true), log id: 2e1a74a >2014-07-29 17:07:27,339 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-8-thread-48) [9a2e425] VDS rack is spm and moved from up calling resetIrs. >2014-07-29 17:07:27,340 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-48) [9a2e425] START, ResetIrsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, ignoreStopFailed = false), log id: 19813d4e >2014-07-29 17:07:27,344 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-48) [9a2e425] START, SpmStopVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9), log id: 4dec59c0 >2014-07-29 17:07:27,352 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-48) [9a2e425] SpmStopVDSCommand::Stopping SPM on vds rack, pool id 00000002-0002-0002-0002-0000000001d9 >2014-07-29 17:07:27,745 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-48) [9a2e425] FINISH, SpmStopVDSCommand, log id: 4dec59c0 >2014-07-29 17:07:27,752 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-48) [9a2e425] FINISH, ResetIrsVDSCommand, log id: 19813d4e >2014-07-29 17:07:27,759 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-8-thread-48) [9a2e425] FINISH, SetVdsStatusVDSCommand, log id: 2e1a74a >2014-07-29 17:07:27,816 INFO [org.ovirt.engine.core.bll.MaintenanceNumberOfVdssCommand] (org.ovirt.thread.pool-8-thread-48) [9a2e425] Lock freed to object EngineLock [exclusiveLocks= , sharedLocks= key: 00000002-0002-0002-0002-0000000001d9 value: POOL >] >2014-07-29 17:07:27,853 INFO [org.ovirt.engine.core.bll.MaintenanceVdsCommand] (org.ovirt.thread.pool-8-thread-48) [9a2e425] Running command: MaintenanceVdsCommand internal: true. Entities affected : ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS >2014-07-29 17:07:27,864 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-48) [9a2e425] Correlation ID: 9a2e425, Job ID: a3fe84e8-7989-41b1-8645-4e4a8bc50e02, Call Stack: null, Custom Event ID: -1, Message: Host rack was switched to Maintenance mode by admin. >2014-07-29 17:07:29,971 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-34) Updated vds status from Preparing for Maintenance to Maintenance in database, vds = 17f7fe56-5f32-40ef-b098-611b84484e0b : rack >2014-07-29 17:07:29,995 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-46) Clearing cache of pool: 00000002-0002-0002-0002-0000000001d9 for problematic entities of VDS: rack. >2014-07-29 17:07:29,997 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-46) Removing vds [17f7fe56-5f32-40ef-b098-611b84484e0b] from the domain in maintenance cache >2014-07-29 17:07:29,997 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-46) Removing host(s) [17f7fe56-5f32-40ef-b098-611b84484e0b] from hosts unseen domain report cache >2014-07-29 17:07:30,002 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-34) START, DisconnectStoragePoolVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, vds_spm_id = 1), log id: 3833a4fb >2014-07-29 17:07:33,765 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-34) FINISH, DisconnectStoragePoolVDSCommand, log id: 3833a4fb >2014-07-29 17:07:33,769 INFO [org.ovirt.engine.core.bll.storage.DisconnectHostFromStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-34) [5484a050] Running command: DisconnectHostFromStoragePoolServersCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-07-29 17:07:33,778 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-34) [5484a050] START, DisconnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, storageType = NFS, connectionList = [{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 665d8683 >2014-07-29 17:07:33,970 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-34) [5484a050] FINISH, DisconnectStorageServerVDSCommand, return: {5083f370-af09-457b-9b73-3440e8b65a07=0}, log id: 665d8683 >2014-07-29 17:07:37,269 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-56) [738ebb7b] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Active to Unknown, reason : no reporting hosts >2014-07-29 17:07:37,275 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-56) [65d9de58] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-07-29 17:07:37,282 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-56) [65d9de58] Correlation ID: 65d9de58, Call Stack: null, Custom Event ID: -1, Message: Invalid status on Data Center Default. Setting status to Non Responsive. >2014-07-29 17:15:45,527 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (ajp--127.0.0.1-8702-11) [c4e13d9] Lock Acquired to object EngineLock [exclusiveLocks= key: 17f7fe56-5f32-40ef-b098-611b84484e0b value: VDS >, sharedLocks= ] >2014-07-29 17:15:45,554 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (org.ovirt.thread.pool-8-thread-31) [c4e13d9] Running command: ActivateVdsCommand internal: false. Entities affected : ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS >2014-07-29 17:15:45,556 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (org.ovirt.thread.pool-8-thread-31) [c4e13d9] Before acquiring lock in order to prevent monitoring for host rack from data-center Default >2014-07-29 17:15:45,557 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (org.ovirt.thread.pool-8-thread-31) [c4e13d9] Lock acquired, from now a monitoring of host will be skipped for host rack from data-center Default >2014-07-29 17:15:45,566 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-8-thread-31) [c4e13d9] START, SetVdsStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 6b5ec9da >2014-07-29 17:15:45,572 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-8-thread-31) [c4e13d9] FINISH, SetVdsStatusVDSCommand, log id: 6b5ec9da >2014-07-29 17:15:45,580 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (org.ovirt.thread.pool-8-thread-31) [c4e13d9] Activate finished. Lock released. Monitoring can run now for host rack from data-center Default >2014-07-29 17:15:45,587 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-31) [c4e13d9] Correlation ID: c4e13d9, Job ID: 7bc35737-4f02-4b77-a702-ebba530509e8, Call Stack: null, Custom Event ID: -1, Message: Host rack was activated by admin. >2014-07-29 17:15:45,595 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (org.ovirt.thread.pool-8-thread-31) [c4e13d9] Lock freed to object EngineLock [exclusiveLocks= key: 17f7fe56-5f32-40ef-b098-611b84484e0b value: VDS >, sharedLocks= ] >2014-07-29 17:15:46,541 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-8) START, GetHardwareInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]), log id: 5ccb869f >2014-07-29 17:15:46,589 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-8) FINISH, GetHardwareInfoVDSCommand, log id: 5ccb869f >2014-07-29 17:15:46,620 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-8) [ba69bba] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS >2014-07-29 17:15:46,671 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-8) [f062e4c] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-07-29 17:15:46,678 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-8) [7cbab491] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-07-29 17:15:46,689 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-8) [7cbab491] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, storageType = NFS, connectionList = [{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 74d27179 >2014-07-29 17:15:46,788 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-8) [7cbab491] FINISH, ConnectStorageServerVDSCommand, return: {5083f370-af09-457b-9b73-3440e8b65a07=0}, log id: 74d27179 >2014-07-29 17:15:46,789 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-8) [7cbab491] Host rack storage connection was succeeded >2014-07-29 17:15:46,796 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-9) START, ConnectStoragePoolVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, masterVersion = 1), log id: 6384ea71 >2014-07-29 17:16:49,795 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-9) FINISH, ConnectStoragePoolVDSCommand, log id: 6384ea71 >2014-07-29 17:16:49,808 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-8) [7cbab491] START, SetMOMPolicyParametersVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 6a2dd8 >2014-07-29 17:16:49,902 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-8) [7cbab491] FINISH, SetMOMPolicyParametersVDSCommand, log id: 6a2dd8 >2014-07-29 17:16:49,910 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-8) [7cbab491] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host rack was set to Up. >2014-07-29 17:16:49,916 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-8) [7cbab491] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host rack. >2014-07-29 17:16:49,939 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-8) [2e41028] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS >2014-07-29 17:16:58,140 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-43) hostFromVds::selectedVds - rack, spmStatus Free, storage pool Default >2014-07-29 17:16:58,146 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-43) starting spm on vds rack, storage pool Default, prevId -1, LVER -1 >2014-07-29 17:16:58,148 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-43) START, SpmStartVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, prevId=-1, prevLVER=-1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 75df2592 >2014-07-29 17:16:58,196 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-43) spmStart polling started: taskId = 4680fed7-f396-4eaf-a348-8683a11b37ab >2014-07-29 17:16:59,202 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-43) spmStart polling ended: taskId = 4680fed7-f396-4eaf-a348-8683a11b37ab task status = finished >2014-07-29 17:16:59,261 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-43) spmStart polling ended, spm status: SPM >2014-07-29 17:16:59,263 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-43) START, HSMClearTaskVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, taskId=4680fed7-f396-4eaf-a348-8683a11b37ab), log id: 7995adce >2014-07-29 17:16:59,308 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-43) FINISH, HSMClearTaskVDSCommand, log id: 7995adce >2014-07-29 17:16:59,309 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-43) FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@3d1df7c3, log id: 75df2592 >2014-07-29 17:16:59,313 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-43) Initialize Irs proxy from vds: 10.34.62.208 >2014-07-29 17:16:59,319 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-43) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208). >2014-07-29 17:16:59,331 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-38) START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 6b6c8d77 >2014-07-29 17:16:59,383 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-38) -- executeIrsBrokerCommand: Attempting on storage pool 00000002-0002-0002-0002-0000000001d9 >2014-07-29 17:16:59,385 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-38) START, HSMGetAllTasksInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 54df2b88 >2014-07-29 17:16:59,390 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-38) FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 54df2b88 >2014-07-29 17:16:59,391 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-38) FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 6b6c8d77 >2014-07-29 17:16:59,392 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (org.ovirt.thread.pool-8-thread-38) Discovered no tasks on Storage Pool Default >2014-07-29 17:19:08,248 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp--127.0.0.1-8702-6) Running command: LoginUserCommand internal: false. >2014-07-29 17:19:08,253 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-07-29 17:19:08,255 ERROR [org.ovirt.engine.core.aaa.filters.RestApiSessionMgmtFilter] (ajp--127.0.0.1-8702-6) Session-TTL header was not passed. Not setting TTL value >2014-07-29 17:23:26,834 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp--127.0.0.1-8702-3) Running command: LoginUserCommand internal: false. >2014-07-29 17:23:26,844 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-3) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-07-29 17:23:26,845 ERROR [org.ovirt.engine.core.aaa.filters.RestApiSessionMgmtFilter] (ajp--127.0.0.1-8702-3) Session-TTL header was not passed. Not setting TTL value >2014-07-29 17:23:54,993 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-4) Principal admin::internal synchronized >2014-07-29 17:24:01,161 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-74) Attempting to update VMs/Templates Ovf. >2014-07-29 17:25:23,674 ERROR [org.ovirt.engine.core.aaa.filters.RestApiSessionMgmtFilter] (ajp--127.0.0.1-8702-1) Session-TTL header was not passed. Not setting TTL value >2014-07-29 17:27:07,853 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp--127.0.0.1-8702-1) Running command: LoginUserCommand internal: false. >2014-07-29 17:27:07,860 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-1) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-07-29 17:27:07,862 ERROR [org.ovirt.engine.core.aaa.filters.RestApiSessionMgmtFilter] (ajp--127.0.0.1-8702-1) Session-TTL header was not passed. Not setting TTL value >2014-07-29 18:05:15,462 WARN [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-8-thread-20) Executing a command: java.util.concurrent.FutureTask , but note that there are 1 tasks in the queue. >2014-07-29 18:23:55,000 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-23) Principal admin::internal synchronized >2014-07-29 18:24:01,189 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-18) Attempting to update VMs/Templates Ovf. >2014-07-29 18:30:45,659 WARN [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-8-thread-22) Executing a command: java.util.concurrent.FutureTask , but note that there are 1 tasks in the queue. >2014-07-29 19:23:55,005 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-18) Principal admin::internal synchronized >2014-07-29 19:24:01,218 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-5) Attempting to update VMs/Templates Ovf. >2014-07-29 20:23:55,012 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-99) Principal admin::internal synchronized >2014-07-29 20:24:01,230 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-57) [cd4c3ce] Attempting to update VMs/Templates Ovf. >2014-07-29 21:23:55,019 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-2) Principal admin::internal synchronized >2014-07-29 21:24:01,244 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-25) Attempting to update VMs/Templates Ovf. >2014-07-29 22:23:55,027 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-18) Principal admin::internal synchronized >2014-07-29 22:24:01,270 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-71) Attempting to update VMs/Templates Ovf. >2014-07-29 22:45:46,731 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-42) domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 in problem. vds: rack >2014-07-29 22:47:13,172 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-55) Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2014-07-29 22:47:13,176 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-55) [296dca52] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-07-29 22:47:13,179 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-55) [296dca52] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Active to Unknown, reason : null >2014-07-29 22:47:13,192 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-55) [296dca52] Correlation ID: 296dca52, Call Stack: null, Custom Event ID: -1, Message: Invalid status on Data Center Default. Setting Data Center status to Non Responsive (On host rack, Error: Cannot access master storage domain). >2014-07-29 22:49:13,220 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-55) [296dca52] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2014-07-29 22:49:13,222 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-55) [296dca52] hostFromVds::selectedVds - rack, spmStatus returned null! >2014-07-29 22:49:13,224 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-55) [296dca52] IrsBroker::Failed::GetStoragePoolInfoVDS >2014-07-29 22:49:13,225 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-55) [296dca52] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2014-07-29 22:49:13,259 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-38) [70fe0194] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage >2014-07-29 22:49:13,283 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-38) [70fe0194] START, ResetIrsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, ignoreStopFailed = true), log id: 139b7b5b >2014-07-29 22:49:13,287 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-38) [70fe0194] START, SpmStopVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9), log id: 5f9fb08c >2014-07-29 22:49:13,293 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-38) [70fe0194] SpmStopVDSCommand::Stopping SPM on vds rack, pool id 00000002-0002-0002-0002-0000000001d9 >2014-07-29 22:50:00,005 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-25) Autorecovering 1 storage domains >2014-07-29 22:50:00,006 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-25) Autorecovering storage domains id: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 >2014-07-29 22:50:00,008 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-25) [8cb28d1] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage >2014-07-29 22:50:00,009 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-25) [8cb28d1] ConnectDomainToStorage. Before Connect all hosts to pool. Time:7/29/14 10:50 PM >2014-07-29 22:50:00,019 INFO [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-8-thread-31) [7d34a6fa] Running command: ConnectStorageToVdsCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2014-07-29 22:50:00,030 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-31) [7d34a6fa] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 58a81b51 >2014-07-29 22:51:13,190 WARN [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-38) [70fe0194] could not stop spm of pool 00000002-0002-0002-0002-0000000001d9 on vds 17f7fe56-5f32-40ef-b098-611b84484e0b - reason: org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcRunTimeException: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException >2014-07-29 22:51:13,192 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-38) [70fe0194] FINISH, SpmStopVDSCommand, log id: 5f9fb08c >2014-07-29 22:51:13,199 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-31) [7d34a6fa] Command ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-07-29 22:51:13,202 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ListVDSCommand] (DefaultQuartzScheduler_Worker-64) Command ListVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-07-29 22:51:13,201 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-84) [7ef2d911] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: VDSNetworkException: org.apache.commons.httpclient.NoHttpResponseException: The server 10.34.62.208 failed to respond >2014-07-29 22:51:13,207 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-38) [70fe0194] FINISH, ResetIrsVDSCommand, log id: 139b7b5b >2014-07-29 22:51:13,206 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-31) [7d34a6fa] FINISH, ConnectStorageServerVDSCommand, log id: 58a81b51 >2014-07-29 22:51:13,206 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-64) Host rack is not responding. It will stay in Connecting state for a grace period of $80 seconds and after that an attempt to fence the host will be issued. >2014-07-29 22:51:13,208 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-84) [7ef2d911] hostFromVds::selectedVds - rack, spmStatus returned null! >2014-07-29 22:51:13,212 ERROR [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-8-thread-31) [7d34a6fa] Command org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-07-29 22:51:13,213 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-64) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:406) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.fetchRunningVms(VdsUpdateRunTimeInfo.java:971) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVmStats(VdsUpdateRunTimeInfo.java:920) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:638) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:235) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) [:1.7.0_55] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_55] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_55] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_55] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_55] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_55] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_55] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_55] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy82.list(Unknown Source) > at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source) [:1.7.0_55] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_55] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_55] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_55] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_55] > >2014-07-29 22:51:13,230 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-38) [70fe0194] Correlation ID: 70fe0194, Job ID: 8663b09a-7f30-48d3-8746-f3236bd6df15, Call Stack: null, Custom Event ID: -1, Message: Failed to Reconstruct Master Domain for Data Center Default. >2014-07-29 22:51:13,225 ERROR [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-8-thread-31) [7d34a6fa] Transaction rolled-back for command: org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand. >2014-07-29 22:51:13,257 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-25) [8cb28d1] ConnectDomainToStorage. After Connect all hosts to pool. Time:7/29/14 10:51 PM >2014-07-29 22:51:13,262 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-48) [296dca52] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue >2014-07-29 22:51:13,264 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-46) starting processDomainRecovery for domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 >2014-07-29 22:51:13,278 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-64) Failed to refresh VDS , vds = 17f7fe56-5f32-40ef-b098-611b84484e0b : rack, VDS Network Error, continuing. >java.net.ConnectException: Connection refused >2014-07-29 22:51:13,323 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-46) Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 was reported by all hosts in status UP as problematic. Not moving the domain to NonOperational because it is being reconstructed now. >2014-07-29 22:51:13,342 WARN [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-46) [e344fc9] CanDoAction of action ReconstructMasterDomain failed. Reasons:VAR__ACTION__RECONSTRUCT_MASTER,VAR__TYPE__STORAGE__DOMAIN,ACTION_TYPE_FAILED_NO_VDS_IN_POOL >2014-07-29 22:51:13,349 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-46) [e344fc9] Removing vds [] from the domain in maintenance cache >2014-07-29 22:51:13,350 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-46) [e344fc9] Removing host(s) [] from hosts unseen domain report cache >2014-07-29 22:51:13,350 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-48) [296dca52] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue >2014-07-29 22:51:16,553 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-68) START, GetHardwareInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]), log id: 19ed538b >2014-07-29 22:51:16,605 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-68) FINISH, GetHardwareInfoVDSCommand, log id: 19ed538b >2014-07-29 22:51:16,630 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-68) [142ed9c5] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS >2014-07-29 22:51:16,724 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (DefaultQuartzScheduler_Worker-68) [142ed9c5] START, DestroyVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vmId=78722009-2727-4715-91bf-dbbb33ce111e, force=false, secondsToWait=0, gracefully=false, reason=), log id: 3758779d >2014-07-29 22:51:16,774 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (DefaultQuartzScheduler_Worker-68) [142ed9c5] FINISH, DestroyVDSCommand, log id: 3758779d >2014-07-29 22:51:16,782 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-68) [5801dc91] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-07-29 22:51:16,789 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-68) [36555a0f] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-07-29 22:51:16,795 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-68) [36555a0f] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, storageType = NFS, connectionList = [{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 15c00120 >2014-07-29 22:51:16,848 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-68) [36555a0f] FINISH, ConnectStorageServerVDSCommand, return: {5083f370-af09-457b-9b73-3440e8b65a07=0}, log id: 15c00120 >2014-07-29 22:51:16,850 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-68) [36555a0f] Host rack storage connection was succeeded >2014-07-29 22:51:16,857 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-3) START, ConnectStoragePoolVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, masterVersion = 1), log id: 6892e72a >2014-07-29 22:51:23,217 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-74) Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Inactive to Unknown, reason : no reporting hosts >2014-07-29 22:52:17,911 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-3) FINISH, ConnectStoragePoolVDSCommand, log id: 6892e72a >2014-07-29 22:52:17,924 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-68) [36555a0f] START, SetMOMPolicyParametersVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 7083f781 >2014-07-29 22:52:18,018 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-68) [36555a0f] FINISH, SetMOMPolicyParametersVDSCommand, log id: 7083f781 >2014-07-29 22:52:18,028 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-68) [36555a0f] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host rack was set to Up. >2014-07-29 22:52:18,034 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-68) [36555a0f] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host rack. >2014-07-29 22:52:18,061 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-68) [1fbe99e6] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS >2014-07-29 22:52:23,517 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-24) hostFromVds::selectedVds - rack, spmStatus Free, storage pool Default >2014-07-29 22:52:23,524 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-24) starting spm on vds rack, storage pool Default, prevId -1, LVER -1 >2014-07-29 22:52:23,526 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-24) START, SpmStartVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, prevId=-1, prevLVER=-1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 6a4b5b4e >2014-07-29 22:52:23,574 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-24) spmStart polling started: taskId = 7961f177-aa71-4305-ad73-4952dd174589 >2014-07-29 22:52:26,589 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-24) spmStart polling ended: taskId = 7961f177-aa71-4305-ad73-4952dd174589 task status = finished >2014-07-29 22:52:26,645 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-24) spmStart polling ended, spm status: SPM >2014-07-29 22:52:26,648 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-24) START, HSMClearTaskVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, taskId=7961f177-aa71-4305-ad73-4952dd174589), log id: 45f0eee3 >2014-07-29 22:52:26,693 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-24) FINISH, HSMClearTaskVDSCommand, log id: 45f0eee3 >2014-07-29 22:52:26,694 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-24) FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@394ceed4, log id: 6a4b5b4e >2014-07-29 22:52:26,698 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-24) Initialize Irs proxy from vds: 10.34.62.208 >2014-07-29 22:52:26,704 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-24) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208). >2014-07-29 22:52:26,716 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-9) START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 34d9648 >2014-07-29 22:52:27,282 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-9) -- executeIrsBrokerCommand: Attempting on storage pool 00000002-0002-0002-0002-0000000001d9 >2014-07-29 22:52:27,284 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-9) START, HSMGetAllTasksInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 4e760c >2014-07-29 22:52:27,290 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-9) FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 4e760c >2014-07-29 22:52:27,290 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-9) FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 34d9648 >2014-07-29 22:52:27,291 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (org.ovirt.thread.pool-8-thread-9) Discovered no tasks on Storage Pool Default >2014-07-29 23:01:14,482 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-13) [4dca0641] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage domain 35 experienced a high latency of 5.85743 seconds from host rack. This may cause performance and functional issues. Please consult your Storage Administrator. >2014-07-29 23:23:55,036 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-67) [2f0280a3] Principal admin::internal synchronized >2014-07-29 23:24:01,284 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-27) Attempting to update VMs/Templates Ovf. >2014-07-29 23:36:53,513 WARN [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-8-thread-4) Executing a command: java.util.concurrent.FutureTask , but note that there are 1 tasks in the queue. >2014-07-30 00:23:55,044 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-77) Principal admin::internal synchronized >2014-07-30 00:24:01,314 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-83) Attempting to update VMs/Templates Ovf. >2014-07-30 01:23:55,050 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-71) Principal admin::internal synchronized >2014-07-30 01:24:01,348 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-73) Attempting to update VMs/Templates Ovf. >2014-07-30 02:23:55,056 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-98) [5aa74089] Principal admin::internal synchronized >2014-07-30 02:24:01,382 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-84) [7ef2d911] Attempting to update VMs/Templates Ovf. >2014-07-30 03:23:55,062 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-6) [4494a716] Principal admin::internal synchronized >2014-07-30 03:24:01,394 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-94) Attempting to update VMs/Templates Ovf. >2014-07-30 03:35:35,000 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-46) Start deleteAgedOutAuditLogs >2014-07-30 03:35:35,001 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-14) Start deleteAgedOutCommandEntities >2014-07-30 03:35:35,006 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-14) Finished deleteAgedOutCommandEntities >2014-07-30 03:35:35,006 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-46) Finished deleteAgedOutAuditLogs >2014-07-30 04:23:55,069 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-70) Principal admin::internal synchronized >2014-07-30 04:24:01,409 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-83) Attempting to update VMs/Templates Ovf. >2014-07-30 05:01:16,631 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-25) domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 in problem. vds: rack >2014-07-30 05:02:52,612 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-44) [60dd31b0] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2014-07-30 05:02:52,618 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-44) [29bdffa0] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-07-30 05:02:52,621 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-44) [29bdffa0] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Active to Unknown, reason : null >2014-07-30 05:02:52,638 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-44) [29bdffa0] Correlation ID: 29bdffa0, Call Stack: null, Custom Event ID: -1, Message: Invalid status on Data Center Default. Setting Data Center status to Non Responsive (On host rack, Error: Cannot access master storage domain). >2014-07-30 05:04:52,662 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-44) [29bdffa0] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2014-07-30 05:04:52,665 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-44) [29bdffa0] hostFromVds::selectedVds - rack, spmStatus returned null! >2014-07-30 05:04:52,667 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-44) [29bdffa0] IrsBroker::Failed::GetStoragePoolInfoVDS >2014-07-30 05:04:52,669 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-44) [29bdffa0] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2014-07-30 05:04:52,705 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-16) [47196bb9] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage >2014-07-30 05:04:52,723 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-16) [47196bb9] START, ResetIrsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, ignoreStopFailed = true), log id: b4ea484 >2014-07-30 05:04:52,727 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-16) [47196bb9] START, SpmStopVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9), log id: 4920ce7b >2014-07-30 05:04:52,733 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-16) [47196bb9] SpmStopVDSCommand::Stopping SPM on vds rack, pool id 00000002-0002-0002-0002-0000000001d9 >2014-07-30 05:05:00,006 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-20) Autorecovering 1 storage domains >2014-07-30 05:05:00,008 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-20) Autorecovering storage domains id: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 >2014-07-30 05:05:00,011 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-20) [21e62018] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage >2014-07-30 05:05:00,015 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-20) [21e62018] ConnectDomainToStorage. Before Connect all hosts to pool. Time:7/30/14 5:05 AM >2014-07-30 05:05:00,027 INFO [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-8-thread-25) [6cfe75d0] Running command: ConnectStorageToVdsCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2014-07-30 05:05:00,033 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-25) [6cfe75d0] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 465107e2 >2014-07-30 05:06:00,045 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-25) [6cfe75d0] FINISH, ConnectStorageServerVDSCommand, return: {5083f370-af09-457b-9b73-3440e8b65a07=0}, log id: 465107e2 >2014-07-30 05:06:00,048 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-20) [21e62018] ConnectDomainToStorage. After Connect all hosts to pool. Time:7/30/14 5:06 AM >2014-07-30 05:06:52,635 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-97) Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: VDSNetworkException: org.apache.commons.httpclient.NoHttpResponseException: The server 10.34.62.208 failed to respond >2014-07-30 05:06:52,637 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetStatsVDSCommand] (DefaultQuartzScheduler_Worker-8) [2e41028] Command GetStatsVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-07-30 05:06:52,636 WARN [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-16) [47196bb9] could not stop spm of pool 00000002-0002-0002-0002-0000000001d9 on vds 17f7fe56-5f32-40ef-b098-611b84484e0b - reason: org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcRunTimeException: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException >2014-07-30 05:06:52,638 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-97) hostFromVds::selectedVds - rack, spmStatus returned null! >2014-07-30 05:06:52,643 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-8) [2e41028] Host rack is not responding. It will stay in Connecting state for a grace period of $80 seconds and after that an attempt to fence the host will be issued. >2014-07-30 05:06:52,642 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-16) [47196bb9] FINISH, SpmStopVDSCommand, log id: 4920ce7b >2014-07-30 05:06:52,646 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-8) [2e41028] vds::refreshVdsStats Failed getVdsStats, vds = 17f7fe56-5f32-40ef-b098-611b84484e0b : rack, error = VDSNetworkException: java.net.ConnectException: Connection refused >2014-07-30 05:06:52,648 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-8) [2e41028] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsStats(VdsUpdateRunTimeInfo.java:669) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:623) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:235) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) [:1.7.0_55] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_55] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_55] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_55] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_55] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_55] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_55] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_55] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy82.getVdsStats(Unknown Source) > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_55] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_55] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_55] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_55] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_55] > >2014-07-30 05:06:52,656 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-16) [47196bb9] FINISH, ResetIrsVDSCommand, log id: b4ea484 >2014-07-30 05:06:52,680 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-8) [2e41028] Failed to refresh VDS , vds = 17f7fe56-5f32-40ef-b098-611b84484e0b : rack, VDS Network Error, continuing. >java.net.ConnectException: Connection refused >2014-07-30 05:06:52,687 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-16) [47196bb9] Correlation ID: 47196bb9, Job ID: da38ecad-2a3a-4d07-8f38-fab75caaf397, Call Stack: null, Custom Event ID: -1, Message: Failed to Reconstruct Master Domain for Data Center Default. >2014-07-30 05:06:52,694 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-46) [29bdffa0] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue >2014-07-30 05:06:52,697 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-26) starting processDomainRecovery for domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 >2014-07-30 05:06:52,760 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-26) Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 was reported by all hosts in status UP as problematic. Not moving the domain to NonOperational because it is being reconstructed now. >2014-07-30 05:06:52,779 WARN [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-26) [1f774652] CanDoAction of action ReconstructMasterDomain failed. Reasons:VAR__ACTION__RECONSTRUCT_MASTER,VAR__TYPE__STORAGE__DOMAIN,ACTION_TYPE_FAILED_NO_VDS_IN_POOL >2014-07-30 05:06:52,786 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-26) [1f774652] Removing vds [] from the domain in maintenance cache >2014-07-30 05:06:52,787 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-26) [1f774652] Removing host(s) [] from hosts unseen domain report cache >2014-07-30 05:06:52,788 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-46) [29bdffa0] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue >2014-07-30 05:06:55,958 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-58) [3d5a7fcf] START, GetHardwareInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]), log id: 19cb0b01 >2014-07-30 05:06:56,013 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-58) [3d5a7fcf] FINISH, GetHardwareInfoVDSCommand, log id: 19cb0b01 >2014-07-30 05:06:56,036 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-58) [27efeab7] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS >2014-07-30 05:06:56,086 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-58) [15769bf8] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-07-30 05:06:56,094 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-58) [7ec29012] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-07-30 05:06:56,105 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-58) [7ec29012] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, storageType = NFS, connectionList = [{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 1589ef68 >2014-07-30 05:06:56,164 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-58) [7ec29012] FINISH, ConnectStorageServerVDSCommand, return: {5083f370-af09-457b-9b73-3440e8b65a07=0}, log id: 1589ef68 >2014-07-30 05:06:56,166 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-58) [7ec29012] Host rack storage connection was succeeded >2014-07-30 05:06:56,174 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-3) START, ConnectStoragePoolVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, masterVersion = 1), log id: 326756f0 >2014-07-30 05:07:02,650 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-27) Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Inactive to Unknown, reason : no reporting hosts >2014-07-30 05:07:56,536 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-3) FINISH, ConnectStoragePoolVDSCommand, log id: 326756f0 >2014-07-30 05:07:56,550 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-58) [7ec29012] START, SetMOMPolicyParametersVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 704871c5 >2014-07-30 05:07:56,643 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-58) [7ec29012] FINISH, SetMOMPolicyParametersVDSCommand, log id: 704871c5 >2014-07-30 05:07:56,650 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-58) [7ec29012] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host rack was set to Up. >2014-07-30 05:07:56,666 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-58) [7ec29012] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host rack. >2014-07-30 05:07:56,683 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-58) [7b19edd8] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS >2014-07-30 05:08:02,796 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-66) [2734597c] hostFromVds::selectedVds - rack, spmStatus Free, storage pool Default >2014-07-30 05:08:02,802 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-66) [2734597c] starting spm on vds rack, storage pool Default, prevId -1, LVER -1 >2014-07-30 05:08:02,803 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-66) [2734597c] START, SpmStartVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, prevId=-1, prevLVER=-1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 72f55fb4 >2014-07-30 05:08:02,852 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-66) [2734597c] spmStart polling started: taskId = 22041af6-af8d-4930-bc1f-e69d1cdad8c5 >2014-07-30 05:08:04,863 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-66) [2734597c] spmStart polling ended: taskId = 22041af6-af8d-4930-bc1f-e69d1cdad8c5 task status = finished >2014-07-30 05:08:04,919 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-66) [2734597c] spmStart polling ended, spm status: SPM >2014-07-30 05:08:04,922 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-66) [2734597c] START, HSMClearTaskVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, taskId=22041af6-af8d-4930-bc1f-e69d1cdad8c5), log id: 5ae2832f >2014-07-30 05:08:04,966 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-66) [2734597c] FINISH, HSMClearTaskVDSCommand, log id: 5ae2832f >2014-07-30 05:08:04,967 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-66) [2734597c] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@1739ea18, log id: 72f55fb4 >2014-07-30 05:08:04,971 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-66) [2734597c] Initialize Irs proxy from vds: 10.34.62.208 >2014-07-30 05:08:04,978 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-66) [2734597c] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208). >2014-07-30 05:08:04,980 WARN [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-8-thread-20) Executing a command: java.util.concurrent.FutureTask , but note that there are 1 tasks in the queue. >2014-07-30 05:08:04,989 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-20) [2734597c] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 3d9f180f >2014-07-30 05:08:05,012 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-20) [2734597c] -- executeIrsBrokerCommand: Attempting on storage pool 00000002-0002-0002-0002-0000000001d9 >2014-07-30 05:08:05,014 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-20) [2734597c] START, HSMGetAllTasksInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 14aaf8ec >2014-07-30 05:08:05,020 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-20) [2734597c] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 14aaf8ec >2014-07-30 05:08:05,021 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-20) [2734597c] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 3d9f180f >2014-07-30 05:08:05,022 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (org.ovirt.thread.pool-8-thread-20) [2734597c] Discovered no tasks on Storage Pool Default >2014-07-30 05:23:55,076 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-63) Principal admin::internal synchronized >2014-07-30 05:24:01,424 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-60) Attempting to update VMs/Templates Ovf. >2014-07-30 06:23:55,085 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-79) Principal admin::internal synchronized >2014-07-30 06:24:01,437 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-81) Attempting to update VMs/Templates Ovf. >2014-07-30 07:23:55,091 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-7) Principal admin::internal synchronized >2014-07-30 07:24:01,451 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-28) [8ef1638] Attempting to update VMs/Templates Ovf. >2014-07-30 08:23:55,096 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-71) Principal admin::internal synchronized >2014-07-30 08:24:01,466 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-91) Attempting to update VMs/Templates Ovf. >2014-07-30 08:29:38,503 WARN [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-8-thread-23) Executing a command: java.util.concurrent.FutureTask , but note that there are 1 tasks in the queue. >2014-07-30 09:18:13,768 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp--127.0.0.1-8702-4) Running command: LoginUserCommand internal: false. >2014-07-30 09:18:13,773 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-4) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-07-30 09:18:19,166 INFO [org.ovirt.engine.core.bll.LogoutUserCommand] (ajp--127.0.0.1-8702-7) [48432f1c] Running command: LogoutUserCommand internal: false. >2014-07-30 09:18:19,174 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-7) [48432f1c] Correlation ID: 48432f1c, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-07-30 09:23:55,102 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-28) [8ef1638] Principal admin::internal synchronized >2014-07-30 09:24:01,481 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-96) Attempting to update VMs/Templates Ovf. >2014-07-30 09:31:14,716 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-87) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage domain 35 experienced a high latency of 5.12679 seconds from host rack. This may cause performance and functional issues. Please consult your Storage Administrator. >2014-07-30 10:23:55,110 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-30) [dd7f8b1] Principal admin::internal synchronized >2014-07-30 10:24:01,494 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-66) [2734597c] Attempting to update VMs/Templates Ovf. >2014-07-30 10:41:22,810 INFO [org.ovirt.engine.core.bll.LoginBaseCommand] (ajp--127.0.0.1-8702-3) Cant login user "admin" with authentication profile "internal" because the authentication failed. >2014-07-30 10:41:22,817 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-3) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin cannot login, please verify the username and password. >2014-07-30 10:41:22,821 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-3) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin failed to log in. >2014-07-30 10:41:22,822 WARN [org.ovirt.engine.core.bll.LoginAdminUserCommand] (ajp--127.0.0.1-8702-3) CanDoAction of action LoginAdminUser failed. Reasons:USER_FAILED_TO_AUTHENTICATE_WRONG_USERNAME_OR_PASSWORD >2014-07-30 10:41:27,478 INFO [org.ovirt.engine.core.bll.LoginAdminUserCommand] (ajp--127.0.0.1-8702-5) Running command: LoginAdminUserCommand internal: false. >2014-07-30 10:41:27,482 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-5) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-07-30 10:41:27,527 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp--127.0.0.1-8702-2) Running command: LoginUserCommand internal: false. >2014-07-30 10:42:31,374 INFO [org.ovirt.engine.core.bll.storage.AddStorageServerConnectionCommand] (ajp--127.0.0.1-8702-5) [29002c1] Lock Acquired to object EngineLock [exclusiveLocks= key: 10.34.63.204:/home/iso/shared value: STORAGE_CONNECTION >, sharedLocks= ] >2014-07-30 10:42:31,386 INFO [org.ovirt.engine.core.bll.storage.AddStorageServerConnectionCommand] (ajp--127.0.0.1-8702-5) [29002c1] Running command: AddStorageServerConnectionCommand internal: false. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2014-07-30 10:42:31,389 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (ajp--127.0.0.1-8702-5) [29002c1] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: null, connection: 10.34.63.204:/home/iso/shared, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 3370d1f8 >2014-07-30 10:42:31,464 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (ajp--127.0.0.1-8702-5) [29002c1] FINISH, ConnectStorageServerVDSCommand, return: {00000000-0000-0000-0000-000000000000=0}, log id: 3370d1f8 >2014-07-30 10:42:31,474 INFO [org.ovirt.engine.core.bll.storage.AddStorageServerConnectionCommand] (ajp--127.0.0.1-8702-5) [29002c1] Lock freed to object EngineLock [exclusiveLocks= key: 10.34.63.204:/home/iso/shared value: STORAGE_CONNECTION >, sharedLocks= ] >2014-07-30 10:42:31,560 INFO [org.ovirt.engine.core.bll.storage.GetExistingStorageDomainListQuery] (ajp--127.0.0.1-8702-7) START, GetExistingStorageDomainListQuery(refresh: true, filtered: false), log id: 286ff6cf >2014-07-30 10:42:31,566 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetStorageDomainsListVDSCommand] (ajp--127.0.0.1-8702-7) START, HSMGetStorageDomainsListVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId=00000000-0000-0000-0000-000000000000, storageType=NFS, storageDomainType=ISO, path=10.34.63.204:/home/iso/shared), log id: 668150fa >2014-07-30 10:43:32,047 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetStorageDomainsListVDSCommand] (ajp--127.0.0.1-8702-7) FINISH, HSMGetStorageDomainsListVDSCommand, return: [0c78b4d6-ba00-4d3e-9f9f-65c7d5899d71], log id: 668150fa >2014-07-30 10:43:32,162 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetStorageDomainInfoVDSCommand] (ajp--127.0.0.1-8702-7) START, HSMGetStorageDomainInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storageDomainId=0c78b4d6-ba00-4d3e-9f9f-65c7d5899d71), log id: 70465c83 >2014-07-30 10:43:32,175 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetStorageDomainInfoVDSCommand] (ajp--127.0.0.1-8702-7) FINISH, HSMGetStorageDomainInfoVDSCommand, return: <org.ovirt.engine.core.common.businessentities.StorageDomainStatic@440d84d6, 00000002-0002-0002-0002-0000000000a8>, log id: 70465c83 >2014-07-30 10:43:32,177 INFO [org.ovirt.engine.core.bll.storage.GetExistingStorageDomainListQuery] (ajp--127.0.0.1-8702-7) FINISH, GetExistingStorageDomainListQuery, log id: 286ff6cf >2014-07-30 10:43:37,403 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetStorageDomainsListVDSCommand] (ajp--127.0.0.1-8702-4) [1729b243] START, HSMGetStorageDomainsListVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId=00000000-0000-0000-0000-000000000000, storageType=NFS, storageDomainType=ISO, path=), log id: b0b433b >2014-07-30 10:44:37,597 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetStorageDomainsListVDSCommand] (ajp--127.0.0.1-8702-4) [1729b243] FINISH, HSMGetStorageDomainsListVDSCommand, return: [0c78b4d6-ba00-4d3e-9f9f-65c7d5899d71], log id: b0b433b >2014-07-30 10:44:37,600 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetStorageDomainInfoVDSCommand] (ajp--127.0.0.1-8702-4) [1729b243] START, HSMGetStorageDomainInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storageDomainId=0c78b4d6-ba00-4d3e-9f9f-65c7d5899d71), log id: 7ae22fb8 >2014-07-30 10:44:37,613 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetStorageDomainInfoVDSCommand] (ajp--127.0.0.1-8702-4) [1729b243] FINISH, HSMGetStorageDomainInfoVDSCommand, return: <org.ovirt.engine.core.common.businessentities.StorageDomainStatic@440d84d6, 00000002-0002-0002-0002-0000000000a8>, log id: 7ae22fb8 >2014-07-30 10:44:37,622 INFO [org.ovirt.engine.core.bll.storage.AddExistingFileStorageDomainCommand] (ajp--127.0.0.1-8702-4) [1729b243] Running command: AddExistingFileStorageDomainCommand internal: false. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2014-07-30 10:44:37,629 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetStorageDomainStatsVDSCommand] (ajp--127.0.0.1-8702-4) [1729b243] START, GetStorageDomainStatsVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storageDomainId=0c78b4d6-ba00-4d3e-9f9f-65c7d5899d71), log id: 6ca5df6b >2014-07-30 10:44:37,678 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetStorageDomainStatsVDSCommand] (ajp--127.0.0.1-8702-4) [1729b243] FINISH, GetStorageDomainStatsVDSCommand, return: org.ovirt.engine.core.common.businessentities.StorageDomain@56c210af, log id: 6ca5df6b >2014-07-30 10:44:37,692 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-4) [1729b243] Correlation ID: 1729b243, Job ID: 5904ce5c-95d8-4c08-a50b-4df2b86c6343, Call Stack: null, Custom Event ID: -1, Message: Storage Domain str01-iso-shared was added by admin >2014-07-30 10:44:37,870 INFO [org.ovirt.engine.core.bll.storage.AttachStorageDomainToPoolCommand] (ajp--127.0.0.1-8702-2) [26e69bac] Running command: AttachStorageDomainToPoolCommand internal: false. Entities affected : ID: 0c78b4d6-ba00-4d3e-9f9f-65c7d5899d71 Type: Storage >2014-07-30 10:44:37,883 INFO [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-8-thread-30) [33d792b6] Running command: ConnectStorageToVdsCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2014-07-30 10:44:37,891 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-30) [33d792b6] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: 51e002a3-3547-4fbf-b3c9-6a4536cca8a8, connection: 10.34.63.204:/home/iso/shared, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 75cd4335 >2014-07-30 10:44:37,902 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-30) [33d792b6] FINISH, ConnectStorageServerVDSCommand, return: {51e002a3-3547-4fbf-b3c9-6a4536cca8a8=0}, log id: 75cd4335 >2014-07-30 10:44:37,905 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.AttachStorageDomainVDSCommand] (ajp--127.0.0.1-8702-2) [26e69bac] START, AttachStorageDomainVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, storageDomainId = 0c78b4d6-ba00-4d3e-9f9f-65c7d5899d71), log id: 1b2b0aad >2014-07-30 10:44:41,258 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.AttachStorageDomainVDSCommand] (ajp--127.0.0.1-8702-2) [26e69bac] FINISH, AttachStorageDomainVDSCommand, log id: 1b2b0aad >2014-07-30 10:44:41,308 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetImagesListVDSCommand] (ajp--127.0.0.1-8702-2) [26e69bac] START, GetImagesListVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, sdUUID = 0c78b4d6-ba00-4d3e-9f9f-65c7d5899d71), log id: 694af1c2 >2014-07-30 10:44:41,365 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetImagesListVDSCommand] (ajp--127.0.0.1-8702-2) [26e69bac] FINISH, GetImagesListVDSCommand, return: [], log id: 694af1c2 >2014-07-30 10:44:41,380 WARN [org.ovirt.engine.core.bll.storage.AttachStorageDomainToPoolCommand] (ajp--127.0.0.1-8702-2) There are no OVF_STORE disks on storage domain id 0c78b4d6-ba00-4d3e-9f9f-65c7d5899d71 >2014-07-30 10:44:41,385 INFO [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (ajp--127.0.0.1-8702-2) [6ef20677] Lock Acquired to object EngineLock [exclusiveLocks= key: 0c78b4d6-ba00-4d3e-9f9f-65c7d5899d71 value: STORAGE >, sharedLocks= ] >2014-07-30 10:44:41,403 INFO [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (ajp--127.0.0.1-8702-2) [6ef20677] Running command: ActivateStorageDomainCommand internal: true. Entities affected : ID: 0c78b4d6-ba00-4d3e-9f9f-65c7d5899d71 Type: Storage >2014-07-30 10:44:41,409 INFO [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (ajp--127.0.0.1-8702-2) [6ef20677] Lock freed to object EngineLock [exclusiveLocks= key: 0c78b4d6-ba00-4d3e-9f9f-65c7d5899d71 value: STORAGE >, sharedLocks= ] >2014-07-30 10:44:41,410 INFO [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (ajp--127.0.0.1-8702-2) [6ef20677] ActivateStorage Domain. Before Connect all hosts to pool. Time:7/30/14 10:44 AM >2014-07-30 10:44:41,415 INFO [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-8-thread-45) [7ca063e4] Running command: ConnectStorageToVdsCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2014-07-30 10:44:41,421 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-45) [7ca063e4] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: 51e002a3-3547-4fbf-b3c9-6a4536cca8a8, connection: 10.34.63.204:/home/iso/shared, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 61e452eb >2014-07-30 10:44:41,434 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-45) [7ca063e4] FINISH, ConnectStorageServerVDSCommand, return: {51e002a3-3547-4fbf-b3c9-6a4536cca8a8=0}, log id: 61e452eb >2014-07-30 10:44:41,437 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ActivateStorageDomainVDSCommand] (ajp--127.0.0.1-8702-2) [6ef20677] START, ActivateStorageDomainVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, storageDomainId = 0c78b4d6-ba00-4d3e-9f9f-65c7d5899d71), log id: 3a04519a >2014-07-30 10:44:41,596 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ActivateStorageDomainVDSCommand] (ajp--127.0.0.1-8702-2) [6ef20677] FINISH, ActivateStorageDomainVDSCommand, log id: 3a04519a >2014-07-30 10:44:41,597 INFO [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (ajp--127.0.0.1-8702-2) [6ef20677] ActivateStorage Domain. After Connect all hosts to pool. Time:7/30/14 10:44 AM >2014-07-30 10:44:41,608 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-1) START, ConnectStoragePoolVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, masterVersion = 1), log id: 409841cf >2014-07-30 10:44:56,319 INFO [org.ovirt.engine.core.bll.AddVmFromScratchCommand] (ajp--127.0.0.1-8702-4) [78d048a] Lock Acquired to object EngineLock [exclusiveLocks= key: ovirtnode value: VM_NAME >, sharedLocks= ] >2014-07-30 10:44:56,379 INFO [org.ovirt.engine.core.bll.AddVmFromScratchCommand] (ajp--127.0.0.1-8702-4) [78d048a] Running command: AddVmFromScratchCommand internal: false. Entities affected : ID: 00000001-0001-0001-0001-0000000002c2 Type: VdsGroups >2014-07-30 10:44:56,522 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-4) [78d048a] Correlation ID: 78d048a, Job ID: 804cbe8f-16a6-42ef-b810-2fef73e7462a, Call Stack: null, Custom Event ID: -1, Message: VM ovirtnode was created by admin. >2014-07-30 10:44:56,528 INFO [org.ovirt.engine.core.bll.AddVmFromScratchCommand] (ajp--127.0.0.1-8702-4) [78d048a] Lock freed to object EngineLock [exclusiveLocks= key: ovirtnode value: VM_NAME >, sharedLocks= ] >2014-07-30 10:44:57,280 INFO [org.ovirt.engine.core.bll.network.vm.AddVmInterfaceCommand] (ajp--127.0.0.1-8702-7) [5e88f035] Running command: AddVmInterfaceCommand internal: false. Entities affected : ID: 9042bb21-3f27-48bb-ab26-c0d8b18c840d Type: VM, ID: 0000000e-000e-000e-000e-0000000003f5 Type: VnicProfile >2014-07-30 10:44:57,319 INFO [org.ovirt.engine.core.bll.network.vm.ActivateDeactivateVmNicCommand] (ajp--127.0.0.1-8702-7) [7c1ea83c] Running command: ActivateDeactivateVmNicCommand internal: true. Entities affected : ID: 9042bb21-3f27-48bb-ab26-c0d8b18c840d Type: VM >2014-07-30 10:44:57,347 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-7) [7c1ea83c] Correlation ID: 7c1ea83c, Call Stack: null, Custom Event ID: -1, Message: Network Interface nic1 (VirtIO) was plugged to VM ovirtnode. (User: admin) >2014-07-30 10:44:57,364 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-7) [7c1ea83c] Correlation ID: 5e88f035, Call Stack: null, Custom Event ID: -1, Message: Interface nic1 (VirtIO) was added to VM ovirtnode. (User: admin) >2014-07-30 10:44:57,441 INFO [org.ovirt.engine.core.bll.network.vm.ReorderVmNicsCommand] (ajp--127.0.0.1-8702-3) [17a8e6f6] Running command: ReorderVmNicsCommand internal: false. Entities affected : ID: 9042bb21-3f27-48bb-ab26-c0d8b18c840d Type: VM >2014-07-30 10:45:04,813 INFO [org.ovirt.engine.core.bll.AddDiskCommand] (ajp--127.0.0.1-8702-11) [2126a430] Lock Acquired to object EngineLock [exclusiveLocks= key: 9042bb21-3f27-48bb-ab26-c0d8b18c840d value: VM_DISK_BOOT >, sharedLocks= key: 9042bb21-3f27-48bb-ab26-c0d8b18c840d value: VM >] >2014-07-30 10:45:04,834 INFO [org.ovirt.engine.core.bll.AddDiskCommand] (ajp--127.0.0.1-8702-11) [2126a430] Running command: AddDiskCommand internal: false. Entities affected : ID: 9042bb21-3f27-48bb-ab26-c0d8b18c840d Type: VM, ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage >2014-07-30 10:45:04,879 INFO [org.ovirt.engine.core.bll.AddImageFromScratchCommand] (ajp--127.0.0.1-8702-11) [1f7f6bf1] Running command: AddImageFromScratchCommand internal: true. Entities affected : ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage >2014-07-30 10:45:04,894 INFO [org.ovirt.engine.core.bll.AddImageFromScratchCommand] (ajp--127.0.0.1-8702-11) [1f7f6bf1] Lock freed to object EngineLock [exclusiveLocks= key: 9042bb21-3f27-48bb-ab26-c0d8b18c840d value: VM_DISK_BOOT >, sharedLocks= key: 9042bb21-3f27-48bb-ab26-c0d8b18c840d value: VM >] >2014-07-30 10:45:04,895 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.CreateImageVDSCommand] (ajp--127.0.0.1-8702-11) [1f7f6bf1] START, CreateImageVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, storageDomainId = 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752, imageGroupId = cc7b5e15-0557-460a-af76-7daa89c27d9f, imageSizeInBytes = 10737418240, volumeFormat = RAW, newImageId = 1efc6019-96fd-4b59-8289-6ce7fe14a614, newImageDescription = ), log id: 4b17b6f0 >2014-07-30 10:45:04,899 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.CreateImageVDSCommand] (ajp--127.0.0.1-8702-11) [1f7f6bf1] -- executeIrsBrokerCommand: calling 'createVolume' with two new parameters: description and UUID >2014-07-30 10:45:05,398 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.CreateImageVDSCommand] (ajp--127.0.0.1-8702-11) [1f7f6bf1] FINISH, CreateImageVDSCommand, return: 1efc6019-96fd-4b59-8289-6ce7fe14a614, log id: 4b17b6f0 >2014-07-30 10:45:05,407 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (ajp--127.0.0.1-8702-11) [1f7f6bf1] CommandAsyncTask::Adding CommandMultiAsyncTasks object for command fa8dcdeb-d342-45d0-97e9-6acc112fe2cf >2014-07-30 10:45:05,408 INFO [org.ovirt.engine.core.bll.CommandMultiAsyncTasks] (ajp--127.0.0.1-8702-11) [1f7f6bf1] CommandMultiAsyncTasks::AttachTask: Attaching task 01677897-7f42-4cd2-8303-56109bb9ef14 to command fa8dcdeb-d342-45d0-97e9-6acc112fe2cf. >2014-07-30 10:45:05,417 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (ajp--127.0.0.1-8702-11) [1f7f6bf1] Adding task 01677897-7f42-4cd2-8303-56109bb9ef14 (Parent Command AddDisk, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters), polling hasn't started yet.. >2014-07-30 10:45:05,436 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-11) [1f7f6bf1] Correlation ID: 2126a430, Job ID: d5224cf3-594f-4add-b92f-f1e82f96e8fd, Call Stack: null, Custom Event ID: -1, Message: Add-Disk operation of ovirtnode_Disk1 was initiated on VM ovirtnode by admin. >2014-07-30 10:45:05,437 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (ajp--127.0.0.1-8702-11) [1f7f6bf1] BaseAsyncTask::startPollingTask: Starting to poll task 01677897-7f42-4cd2-8303-56109bb9ef14. >2014-07-30 10:45:13,116 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (DefaultQuartzScheduler_Worker-43) Polling and updating Async Tasks: 1 tasks, 1 tasks to poll now >2014-07-30 10:45:13,123 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (DefaultQuartzScheduler_Worker-43) SPMAsyncTask::PollTask: Polling task 01677897-7f42-4cd2-8303-56109bb9ef14 (Parent Command AddDisk, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) returned status finished, result 'success'. >2014-07-30 10:45:13,130 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (DefaultQuartzScheduler_Worker-43) BaseAsyncTask::OnTaskEndSuccess: Task 01677897-7f42-4cd2-8303-56109bb9ef14 (Parent Command AddDisk, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) ended successfully. >2014-07-30 10:45:13,132 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (DefaultQuartzScheduler_Worker-43) CommandAsyncTask::EndActionIfNecessary: All tasks of command fa8dcdeb-d342-45d0-97e9-6acc112fe2cf has ended -> executing endAction >2014-07-30 10:45:13,133 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (DefaultQuartzScheduler_Worker-43) CommandAsyncTask::endAction: Ending action for 1 tasks (command ID: fa8dcdeb-d342-45d0-97e9-6acc112fe2cf): calling endAction . >2014-07-30 10:45:13,135 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-8-thread-23) CommandAsyncTask::EndCommandAction [within thread] context: Attempting to endAction AddDisk, executionIndex: 0 >2014-07-30 10:45:13,146 INFO [org.ovirt.engine.core.bll.AddDiskCommand] (org.ovirt.thread.pool-8-thread-23) [2126a430] Ending command successfully: org.ovirt.engine.core.bll.AddDiskCommand >2014-07-30 10:45:13,152 INFO [org.ovirt.engine.core.bll.AddImageFromScratchCommand] (org.ovirt.thread.pool-8-thread-23) [1f7f6bf1] Ending command successfully: org.ovirt.engine.core.bll.AddImageFromScratchCommand >2014-07-30 10:45:13,162 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetImageInfoVDSCommand] (org.ovirt.thread.pool-8-thread-23) [1f7f6bf1] START, GetImageInfoVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, storageDomainId = 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752, imageGroupId = cc7b5e15-0557-460a-af76-7daa89c27d9f, imageId = 1efc6019-96fd-4b59-8289-6ce7fe14a614), log id: 707002a5 >2014-07-30 10:45:13,184 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetImageInfoVDSCommand] (org.ovirt.thread.pool-8-thread-23) [1f7f6bf1] FINISH, GetImageInfoVDSCommand, return: org.ovirt.engine.core.common.businessentities.DiskImage@8fd12089, log id: 707002a5 >2014-07-30 10:45:13,201 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-23) Correlation ID: 2126a430, Call Stack: null, Custom Event ID: -1, Message: The disk ovirtnode_Disk1 was successfully added to VM ovirtnode. >2014-07-30 10:45:13,202 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-8-thread-23) CommandAsyncTask::HandleEndActionResult [within thread]: endAction for action type AddDisk completed, handling the result. >2014-07-30 10:45:13,203 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-8-thread-23) CommandAsyncTask::HandleEndActionResult [within thread]: endAction for action type AddDisk succeeded, clearing tasks. >2014-07-30 10:45:13,207 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (org.ovirt.thread.pool-8-thread-23) SPMAsyncTask::ClearAsyncTask: Attempting to clear task 01677897-7f42-4cd2-8303-56109bb9ef14 >2014-07-30 10:45:13,209 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-23) START, SPMClearTaskVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, taskId = 01677897-7f42-4cd2-8303-56109bb9ef14), log id: 1b8d2334 >2014-07-30 10:45:13,211 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-23) START, HSMClearTaskVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, taskId=01677897-7f42-4cd2-8303-56109bb9ef14), log id: 4f02390d >2014-07-30 10:45:13,267 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-23) FINISH, HSMClearTaskVDSCommand, log id: 4f02390d >2014-07-30 10:45:13,268 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-23) FINISH, SPMClearTaskVDSCommand, log id: 1b8d2334 >2014-07-30 10:45:13,275 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (org.ovirt.thread.pool-8-thread-23) BaseAsyncTask::RemoveTaskFromDB: Removed task 01677897-7f42-4cd2-8303-56109bb9ef14 from DataBase >2014-07-30 10:45:13,276 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-8-thread-23) CommandAsyncTask::HandleEndActionResult [within thread]: Removing CommandMultiAsyncTasks object for entity fa8dcdeb-d342-45d0-97e9-6acc112fe2cf >2014-07-30 10:45:16,487 INFO [org.ovirt.engine.core.bll.RunVmCommand] (ajp--127.0.0.1-8702-6) [6026cfc8] Lock Acquired to object EngineLock [exclusiveLocks= key: 9042bb21-3f27-48bb-ab26-c0d8b18c840d value: VM >, sharedLocks= ] >2014-07-30 10:45:16,528 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (ajp--127.0.0.1-8702-6) [6026cfc8] START, IsVmDuringInitiatingVDSCommand( vmId = 9042bb21-3f27-48bb-ab26-c0d8b18c840d), log id: da8cb13 >2014-07-30 10:45:16,529 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (ajp--127.0.0.1-8702-6) [6026cfc8] FINISH, IsVmDuringInitiatingVDSCommand, return: false, log id: da8cb13 >2014-07-30 10:45:16,586 INFO [org.ovirt.engine.core.bll.RunVmCommand] (org.ovirt.thread.pool-8-thread-2) [6026cfc8] Running command: RunVmCommand internal: false. Entities affected : ID: 9042bb21-3f27-48bb-ab26-c0d8b18c840d Type: VM >2014-07-30 10:45:16,622 INFO [org.ovirt.engine.core.vdsbroker.UpdateVmDynamicDataVDSCommand] (org.ovirt.thread.pool-8-thread-2) [6026cfc8] START, UpdateVmDynamicDataVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vmDynamic=org.ovirt.engine.core.common.businessentities.VmDynamic@3f3f6d6b), log id: 4107216f >2014-07-30 10:45:16,637 INFO [org.ovirt.engine.core.vdsbroker.UpdateVmDynamicDataVDSCommand] (org.ovirt.thread.pool-8-thread-2) [6026cfc8] FINISH, UpdateVmDynamicDataVDSCommand, log id: 4107216f >2014-07-30 10:45:16,659 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (org.ovirt.thread.pool-8-thread-2) [6026cfc8] START, CreateVmVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vmId=9042bb21-3f27-48bb-ab26-c0d8b18c840d, vm=VM [ovirtnode]), log id: 410e0697 >2014-07-30 10:45:16,683 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVmFromCloudInitVDSCommand] (org.ovirt.thread.pool-8-thread-2) [6026cfc8] START, CreateVmFromCloudInitVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vmId=9042bb21-3f27-48bb-ab26-c0d8b18c840d, vm=VM [ovirtnode]), log id: 40951fe9 >2014-07-30 10:45:16,725 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (org.ovirt.thread.pool-8-thread-2) [6026cfc8] org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVmFromCloudInitVDSCommand kvmEnable=true,keyboardLayout=en-us,nice=0,displayNetwork=ovirtmgmt,copyPasteEnable=true,timeOffset=0,transparentHugePages=true,vmId=9042bb21-3f27-48bb-ab26-c0d8b18c840d,acpiEnable=true,custom={},spiceSslCipherSuite=DEFAULT,memSize=1024,smp=1,emulatedMachine=rhel6.5.0,vmType=kvm,memGuaranteedSize=1024,display=qxl,smartcardEnable=false,bootMenuEnable=false,spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,susbredir,ssmartcard,smpCoresPerSocket=1,numaTune={mode=preferred},maxVCpus=160,devices=[{shared=false, iface=ide, index=3, specParams={vmPayload={file={openstack/latest/meta_data.json=ewogICJsYXVuY2hfaW5kZXgiIDogIjAiLAogICJhdmFpbGFiaWxpdHlfem9uZSIgOiAibm92YSIs >CiAgIm5ldHdvcmstaW50ZXJmYWNlcyIgOiAiYXV0b1xuIiwKICAibmV0d29ya19jb25maWciIDog >ewogICAgImNvbnRlbnRfcGF0aCIgOiAiL2NvbnRlbnQvMDAwMCIsCiAgICAicGF0aCIgOiAiL2V0 >Yy9uZXR3b3JrL2ludGVyZmFjZXMiCiAgfSwKICAidXVpZCIgOiAiMTRhZjMwMDYtMmJhMC00YWJj >LThlNGQtMjNmNjA5MGMyY2U1IiwKICAibWV0YSIgOiB7CiAgICAiZXNzZW50aWFsIiA6ICJmYWxz >ZSIsCiAgICAicm9sZSIgOiAic2VydmVyIiwKICAgICJkc21vZGUiIDogImxvY2FsIgogIH0KfQ== >, openstack/latest/user_data=I2Nsb3VkLWNvbmZpZwpzc2hfcHdhdXRoOiB0cnVlCmRpc2FibGVfcm9vdDogMApvdXRwdXQ6CiAg >YWxsOiAnPj4gL3Zhci9sb2cvY2xvdWQtaW5pdC1vdXRwdXQubG9nJwpjaHBhc3N3ZDoKICBleHBp >cmU6IGZhbHNlCnJ1bmNtZDoKLSAnc2VkIC1pICcnL15kYXRhc291cmNlX2xpc3Q6IC9kJycgL2V0 >Yy9jbG91ZC9jbG91ZC5jZmc7IGVjaG8gJydkYXRhc291cmNlX2xpc3Q6CiAgWyJOb0Nsb3VkIiwg >IkNvbmZpZ0RyaXZlIl0nJyA+PiAvZXRjL2Nsb3VkL2Nsb3VkLmNmZycK >, openstack/content/0000=YXV0bwo= >}, volId=config-2}}, path=, device=cdrom, type=disk, readonly=true, deviceId=58b5c483-7135-422b-81ae-ef1624176b93}, {specParams={vram=32768, heads=1}, device=qxl, type=video, deviceId=05cedad9-c992-4b63-89e9-3708ee7e3d89}, {shared=false, iface=ide, index=2, specParams={path=}, path=, device=cdrom, type=disk, readonly=true, deviceId=cbd5ab2a-dc25-451c-9407-ab762c3570da}, {shared=false, volumeID=1efc6019-96fd-4b59-8289-6ce7fe14a614, index=0, iface=virtio, imageID=cc7b5e15-0557-460a-af76-7daa89c27d9f, domainID=71f82fcd-4fa6-4f56-b228-f4fa4bd4f752, specParams={}, optional=false, propagateErrors=off, device=disk, poolID=00000002-0002-0002-0002-0000000001d9, format=raw, type=disk, readonly=false, deviceId=cc7b5e15-0557-460a-af76-7daa89c27d9f}, {nicModel=pv, specParams={outbound={}, inbound={}}, macAddr=00:1a:4a:ae:a8:01, device=bridge, linkActive=true, type=interface, filter=vdsm-no-mac-spoofing, network=ovirtmgmt, deviceId=17e210d5-07a0-4c2a-b63d-245a4f01ec21}, {index=0, model=virtio-scsi, specParams={}, device=scsi, type=controller, deviceId=77986732-043a-4105-9c7b-231a79a0db6f}],vmName=ovirtnode,cpuType=Westmere,fileTransferEnable=true >2014-07-30 10:45:16,781 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVmFromCloudInitVDSCommand] (org.ovirt.thread.pool-8-thread-2) [6026cfc8] FINISH, CreateVmFromCloudInitVDSCommand, log id: 40951fe9 >2014-07-30 10:45:16,788 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (org.ovirt.thread.pool-8-thread-2) [6026cfc8] FINISH, CreateVmVDSCommand, return: WaitForLaunch, log id: 410e0697 >2014-07-30 10:45:16,790 INFO [org.ovirt.engine.core.bll.RunVmCommand] (org.ovirt.thread.pool-8-thread-2) [6026cfc8] Lock freed to object EngineLock [exclusiveLocks= key: 9042bb21-3f27-48bb-ab26-c0d8b18c840d value: VM >, sharedLocks= ] >2014-07-30 10:45:16,795 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-2) [6026cfc8] Correlation ID: 6026cfc8, Job ID: 096197b2-7ed6-4ea0-ad29-fd919dca24af, Call Stack: null, Custom Event ID: -1, Message: VM ovirtnode was started by admin (Host: rack). >2014-07-30 10:45:41,891 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-1) FINISH, ConnectStoragePoolVDSCommand, log id: 409841cf >2014-07-30 10:45:41,892 INFO [org.ovirt.engine.core.bll.storage.RefreshPoolSingleAsyncOperation] (org.ovirt.thread.pool-8-thread-1) Refreshed vds rack in pool Default >2014-07-30 10:45:41,894 INFO [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (ajp--127.0.0.1-8702-2) [6ef20677] ActivateStorage Domain. After change storage pool status in vds. Time:7/30/14 10:45 AM >2014-07-30 10:45:41,896 WARN [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-8-thread-29) Executing a command: java.util.concurrent.FutureTask , but note that there are 1 tasks in the queue. >2014-07-30 10:45:41,900 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-29) [6ef20677] START, GetFileStatsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 26fc1628 >2014-07-30 10:45:41,904 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-2) [6ef20677] Correlation ID: 6ef20677, Call Stack: null, Custom Event ID: -1, Message: Storage Domain str01-iso-shared (Data Center Default) was activated by Non interactive user >2014-07-30 10:45:41,919 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-2) [6ef20677] Correlation ID: 26e69bac, Job ID: 747052ef-c6f7-4a4e-8e32-2682e514cb99, Call Stack: null, Custom Event ID: -1, Message: Storage Domain str01-iso-shared was attached to Data Center Default by admin >2014-07-30 10:45:42,415 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-29) [6ef20677] FINISH, GetFileStatsVDSCommand, return: {deepin_2014_final_amd64.iso={status=0, ctime=1404720000.0, size=1581383680}, RHEV-toolsSetup_2.2_52998.iso={status=0, ctime=1403850000.0, size=105551872}, ubuntu-12.04.4-server-amd64.iso={status=0, ctime=1403850000.0, size=711983104}, RHEV-toolsSetup_3.2_15.iso={status=0, ctime=1403850000.0, size=264646656}, ubuntu-12.04-minicd.iso={status=0, ctime=1403850000.0, size=31457280}, rhev-hypervisor6-6.5-20140715.0.iso={status=0, ctime=1405590000.0, size=224395264}, RHEV-toolsSetup_3.1_8.iso={status=0, ctime=1403850000.0, size=220803072}, RHEV-toolsSetup_3.4_6.iso={status=0, ctime=1403850000.0, size=327026688}, virtio-win-1.6.7.iso={status=0, ctime=1403850000.0, size=97478656}, ovirt-guest-tools-3.5-2.iso={status=0, ctime=1403850000.0, size=76455936}, RHEV-toolsSetup_3.3_10.iso={status=0, ctime=1403850000.0, size=295870464}, RHEV-toolsSetup_3.0_34.iso={status=0, ctime=1403850000.0, size=133777408}, ubuntu-12.04-minimal-amd64.iso={status=0, ctime=1403850000.0, size=31457280}, debian-7.4.0-amd64-netinst.iso={status=0, ctime=1403850000.0, size=232783872}, RHEV-toolsSetup_3.4_7.iso={status=0, ctime=1403850000.0, size=327038976}, Windows2008-x64.iso={status=0, ctime=1403850000.0, size=2665758720}, en_windows_server_2012_r2_x64_dvd_2707946.iso={status=0, ctime=1403850000.0, size=4268605440}, elive_2.3.0_beta_hybrid.iso={status=0, ctime=1403850000.0, size=2187083776}, RHEV-toolsSetup_3.2_17.iso={status=0, ctime=1403850000.0, size=261437440}, RHEV-toolsSetup_3.3_7.iso={status=0, ctime=1403850000.0, size=295516160}, RHEV-toolsSetup_3.3_13.iso={status=0, ctime=1403850000.0, size=299448320}, en_windows_8_enterprise_x86_dvd_917587.iso={status=0, ctime=1403850000.0, size=2551154688}, RHEV-toolsSetup_3.2_16.iso={status=0, ctime=1403850000.0, size=261437440}, SLES-11-SP3-DVD-x86_64-GM-DVD2.iso={status=0, ctime=1403850000.0, size=5311318016}, centos.iso={status=0, ctime=1403850000.0, size=4467982336}, en-gb_windows_8.1_enterprise_with_update_x64_dvd_4048611.iso={status=0, ctime=1403850000.0, size=3964362752}, ubuntu-14.04-minimal-amd64.iso={status=0, ctime=1403850000.0, size=38797312}, RHEV-toolsSetup_3.4_1.iso={status=0, ctime=1403850000.0, size=326516736}, Fedora-Live-LXDE-x86_64-20-1.iso={status=0, ctime=1403850000.0, size=663748608}, ubuntu-13.10-server-amd64.iso={status=0, ctime=1403850000.0, size=704643072}, IT_WINXP_PRO_VL_ISO.iso={status=0, ctime=1403850000.0, size=515557376}, RHEV-toolsSetup_3.4_8.iso={status=0, ctime=1403850000.0, size=327120896}, ubuntu-14.04-server-amd64.iso={status=0, ctime=1403850000.0, size=591396864}, RHEV-toolsSetup_3.0_39.iso={status=0, ctime=1403850000.0, size=134195200}, ubuntu-13.10-minimal-amd64.iso={status=0, ctime=1403850000.0, size=37748736}, en_windows_7_enterprise_x86_dvd_x15-70745.iso={status=0, ctime=1403850000.0, size=2400401408}, RHEV-toolsSetup_3.0_37_ic156.iso={status=0, ctime=1403850000.0, size=133894144}, RHEV-toolsSetup_3.1_12.iso={status=0, ctime=1403850000.0, size=224448512}, EnglishWindowsServer2003R2withSP2VLPStandard64bitDISC1.iso={status=0, ctime=1403850000.0, size=652234752}, RHEV-toolsSetup_3.4_2.el6ev.iso={status=0, ctime=1403850000.0, size=326518784}, MUI_Windows_server_2003_r2_cd1.iso={status=0, ctime=1403850000.0, size=638373888}, openSUSE-13.1-NET-x86_64.iso={status=0, ctime=1403850000.0, size=297795584}, openSUSE-12.3-NET-x86_64.iso={status=0, ctime=1403850000.0, size=252706816}, install55.iso={status=0, ctime=1403850000.0, size=242722816}, RHEV-toolsSetup_3.4_4.el6ev.iso={status=0, ctime=1403850000.0, size=326424576}, FreeBSD-9.2-RELEASE-amd64-bootonly.iso={status=0, ctime=1404200000.0, size=172070912}, ubuntu-13.04-server-amd64.iso={status=0, ctime=1403850000.0, size=735051776}, en_windows_7_home_basic_with_sp1_x86_dvd_u_676470.iso={status=0, ctime=1403850000.0, size=2564476928}, es_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-74009.iso={status=0, ctime=1404390000.0, size=619194368}, RHEV-toolsSetup_3.3_6.iso={status=0, ctime=1403850000.0, size=295997440}, ovirt-guest-tools-3.5_5.iso={status=0, ctime=1406200000.0, size=106803200}, rhev-hypervisor6-6.5-20140112.0.iso={status=0, ctime=1403850000.0, size=226492416}, RHEV-toolsSetup_3.3_3.iso={status=0, ctime=1403850000.0, size=295911424}, RHEV-toolsSetup_3.4_5.el6ev.iso={status=0, ctime=1403850000.0, size=326780928}, fakeISO1.iso={status=0, ctime=1403850000.0, size=0}, es_windows_xp_professional_with_service_pack_3_x86_cd_x14-80488.iso={status=0, ctime=1404390000.0, size=619196416}, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso={status=0, ctime=1403850000.0, size=2996799488}, MX-14.2-pae.iso={status=0, ctime=1405060000.0, size=732954624}, RHEV-toolsSetup_2.3_395.iso={status=0, ctime=1403850000.0, size=133640192}, SLES-11-SP3-DVD-i586-GM-DVD1.iso={status=0, ctime=1403850000.0, size=3221225472}, ovirt-node-iso-3.1.0-0.999.999.20140317175137gitba60c851296.el6.iso={status=0, ctime=1406710000.0, size=211812352}, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso={status=0, ctime=1403850000.0, size=617754624}, RHEV-toolsSetup_2.3_359.iso={status=0, ctime=1403850000.0, size=133640192}, RHEV-toolsSetup_3.3_9.iso={status=0, ctime=1403850000.0, size=295872512}, FreeBSD-10.0-RELEASE-amd64-bootonly.iso={status=0, ctime=1404220000.0, size=220153856}, Pinguy_OS_14.04.1-Mini-x86-64.iso={status=0, ctime=1403850000.0, size=1583349760}, RHEV-toolsSetup_2.3_360-fake.iso={status=0, ctime=1403850000.0, size=0}, SLES-11-SP3-DVD-x86_64-GM-DVD1.iso={status=0, ctime=1403850000.0, size=3362783232}, RHEV-toolsSetup_3.0_37.iso={status=0, ctime=1403850000.0, size=133894144}, WANemv2.3.iso={status=0, ctime=1403850000.0, size=398247936}, RHEV-toolsSetup_3.3_14.iso={status=0, ctime=1403850000.0, size=299448320}, RHEV-toolsSetup_2.2_53011.iso={status=0, ctime=1403850000.0, size=105551872}, ovirt-guest-tools-3.4_2.iso={status=0, ctime=1403850000.0, size=76455936}, RHEV-toolsSetup_3.1_14.iso={status=0, ctime=1403850000.0, size=242872320}, ovirt-guest-tools-3.4-2.iso={status=0, ctime=1403850000.0, size=76455936}, es_windows_7_enterprise_with_sp1_x64_dvd_u_677652.iso={status=0, ctime=1404390000.0, size=3110127616}, EnglishWindowsServer2003R2withSP2VLPStandard64bitDISC2.iso={status=0, ctime=1403850000.0, size=175163392}, fakeISO01.iso={status=0, ctime=1403850000.0, size=0}, en_windows_7_enterprise_x64_dvd_x15-70749.iso={status=0, ctime=1403850000.0, size=3121215488}, Windows2008-x86.iso={status=0, ctime=1403850000.0, size=1881141248}, RHEL-7.0-20131222.0-Server-x86_64-dvd1.iso={status=0, ctime=1403850000.0, size=4580179968}, systemrescuecd-x86-4.2.0.iso={status=0, ctime=1404370000.0, size=399329280}, RHEV-toolsSetup_3.2_18.iso={status=0, ctime=1403850000.0, size=261439488}, CentOS-6.5-x86_64-minimal.iso={status=0, ctime=1403850000.0, size=417333248}, RHEV-toolsSetup_3.1_9.iso={status=0, ctime=1403850000.0, size=220809216}, RHEV-toolsSetup_3.3_8.iso={status=0, ctime=1403850000.0, size=295516160}, en_windows_8_enterprise_x64_dvd_917522.iso={status=0, ctime=1403850000.0, size=3490912256}, ubuntu-10.04-minimal-amd64.iso={status=0, ctime=1403850000.0, size=14299136}, elementaryos-stable-amd64.20130810.iso={status=0, ctime=1403850000.0, size=727711744}, qwerty.iso={status=0, ctime=1403850000.0, size=49530880}, RHEV-toolsSetup_3.3_11.iso={status=0, ctime=1403850000.0, size=295870464}, fakeISO02.iso={status=0, ctime=1403850000.0, size=0}, RHEL6.3-20120613.2-Server-x86_64-DVD1.iso={status=0, ctime=1403850000.0, size=3679453184}, Windows2003-r2-std-32-bit.iso={status=0, ctime=1403850000.0, size=600563712}, slax-English-US-7.0.8-x86_64.iso={status=0, ctime=1403850000.0, size=236482560}, en_windows_server_2012_x64_dvd_915478.iso={status=0, ctime=1403850000.0, size=3695179776}}, log id: 26fc1628 >2014-07-30 10:45:42,554 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-29) [6ef20677] START, GetFileStatsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: df24d4 >2014-07-30 10:45:42,788 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-29) [6ef20677] FINISH, GetFileStatsVDSCommand, return: {winxp.vfd={status=0, ctime=1403850000.0, size=1474560}, virtio-win-1.7.1_x86.vfd={status=0, ctime=1403850000.0, size=2949120}, virtio-win-1.7.1_amd64.vfd={status=0, ctime=1403850000.0, size=2949120}, win2003-32-unattended.vfd={status=0, ctime=1403850000.0, size=1474560}, winXP-Pro-SP2-32-unattended.vfd={status=0, ctime=1403850000.0, size=1474560}, win2000-Pro-unattended.vfd={status=0, ctime=1403850000.0, size=1474560}, win2k3x64.vfd={status=0, ctime=1403850000.0, size=1474560}, virtio-win_amd64.vfd={status=0, ctime=1403850000.0, size=2949120}, winXP-x64-unattended.vfd={status=0, ctime=1403850000.0, size=1474560}, virtio-win_x86.vfd={status=0, ctime=1403850000.0, size=2949120}, win2003-x64-unattended.vfd={status=0, ctime=1403850000.0, size=1474560}, floppy55.vfd={status=0, ctime=1403850000.0, size=1474560}, virtio-win-1.6.7_amd64.vfd={status=0, ctime=1403850000.0, size=2949120}}, log id: df24d4 >2014-07-30 10:45:44,923 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-65) VM ovirtnode 9042bb21-3f27-48bb-ab26-c0d8b18c840d moved from WaitForLaunch --> PoweringUp >2014-07-30 10:45:44,924 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVdsCommand] (DefaultQuartzScheduler_Worker-65) START, FullListVdsCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b], vmIds=[9042bb21-3f27-48bb-ab26-c0d8b18c840d]), log id: 63f38ad7 >2014-07-30 10:45:44,933 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVdsCommand] (DefaultQuartzScheduler_Worker-65) FINISH, FullListVdsCommand, return: [{displaySecurePort=5901, kvmEnable=true, nicModel=rtl8139,pv, keyboardLayout=en-us, displayIp=10.34.62.208, pauseCode=NOERR, nice=0, displayNetwork=ovirtmgmt, copyPasteEnable=true, timeOffset=0, transparentHugePages=true, vmId=9042bb21-3f27-48bb-ab26-c0d8b18c840d, acpiEnable=true, custom={}, spiceSslCipherSuite=DEFAULT, memSize=1024, smp=1, displayPort=5900, emulatedMachine=rhel6.5.0, vmType=kvm, status=Up, memGuaranteedSize=1024, display=qxl, pid=22803, smartcardEnable=false, bootMenuEnable=false, spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,susbredir,ssmartcard, numaTune={mode=preferred}, smpCoresPerSocket=1, maxVCpus=160, clientIp=, devices=[Ljava.lang.Object;@a205c85, vmName=ovirtnode, fileTransferEnable=true, cpuType=Westmere}], log id: 63f38ad7 >2014-07-30 10:45:44,946 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-65) Received a memballoon Device without an address when processing VM 9042bb21-3f27-48bb-ab26-c0d8b18c840d devices, skipping device: {alias=balloon0, specParams={model=none}, device=memballoon, type=balloon} >2014-07-30 10:45:44,947 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-65) Received a spice Device without an address when processing VM 9042bb21-3f27-48bb-ab26-c0d8b18c840d devices, skipping device: {port=5900, specParams={spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,susbredir,ssmartcard, keyMap=en-us, displayNetwork=ovirtmgmt, copyPasteEnable=true, displayIp=10.34.62.208}, device=spice, tlsPort=5901, type=graphics} >2014-07-30 10:45:47,666 INFO [org.ovirt.engine.core.bll.SetVmTicketCommand] (ajp--127.0.0.1-8702-3) [4bc8cf97] Running command: SetVmTicketCommand internal: false. Entities affected : ID: 9042bb21-3f27-48bb-ab26-c0d8b18c840d Type: VM >2014-07-30 10:45:47,672 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetVmTicketVDSCommand] (ajp--127.0.0.1-8702-3) [4bc8cf97] START, SetVmTicketVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vmId=9042bb21-3f27-48bb-ab26-c0d8b18c840d, ticket=+2QnNV2WQwoy, validTime=120,m userName=admin, userId=fdfc627c-d875-11e0-90f0-83df133b58cc), log id: 269088d1 >2014-07-30 10:45:47,789 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetVmTicketVDSCommand] (ajp--127.0.0.1-8702-3) [4bc8cf97] FINISH, SetVmTicketVDSCommand, log id: 269088d1 >2014-07-30 10:45:47,799 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-3) [4bc8cf97] Correlation ID: 4bc8cf97, Call Stack: null, Custom Event ID: -1, Message: user admin initiated console session for VM ovirtnode >2014-07-30 10:45:54,200 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-22) [7e0971f4] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin is connected to VM ovirtnode. >2014-07-30 10:46:12,468 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-39) [4a2f7e33] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin got disconnected from VM ovirtnode. >2014-07-30 10:46:16,115 INFO [org.ovirt.engine.core.bll.StopVmCommand] (org.ovirt.thread.pool-8-thread-19) [636d32ad] Running command: StopVmCommand internal: false. Entities affected : ID: 9042bb21-3f27-48bb-ab26-c0d8b18c840d Type: VM >2014-07-30 10:46:16,120 INFO [org.ovirt.engine.core.vdsbroker.DestroyVmVDSCommand] (org.ovirt.thread.pool-8-thread-19) [636d32ad] START, DestroyVmVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vmId=9042bb21-3f27-48bb-ab26-c0d8b18c840d, force=false, secondsToWait=0, gracefully=false, reason=), log id: 289cd342 >2014-07-30 10:46:16,132 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (org.ovirt.thread.pool-8-thread-19) [636d32ad] START, DestroyVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vmId=9042bb21-3f27-48bb-ab26-c0d8b18c840d, force=false, secondsToWait=0, gracefully=false, reason=), log id: 7c3ef896 >2014-07-30 10:46:16,565 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (org.ovirt.thread.pool-8-thread-19) [636d32ad] FINISH, DestroyVDSCommand, log id: 7c3ef896 >2014-07-30 10:46:16,573 INFO [org.ovirt.engine.core.vdsbroker.DestroyVmVDSCommand] (org.ovirt.thread.pool-8-thread-19) [636d32ad] FINISH, DestroyVmVDSCommand, return: Down, log id: 289cd342 >2014-07-30 10:46:16,580 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-19) [636d32ad] Correlation ID: 636d32ad, Job ID: 6ec8048d-f065-4954-a4c6-ca220f2e9e26, Call Stack: null, Custom Event ID: -1, Message: VM ovirtnode powered off by admin (Host: rack) (Reason: Not Specified). >2014-07-30 10:46:16,586 INFO [org.ovirt.engine.core.bll.ProcessDownVmCommand] (org.ovirt.thread.pool-8-thread-36) [12ece3d4] Running command: ProcessDownVmCommand internal: true. >2014-07-30 10:46:29,059 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (DefaultQuartzScheduler_Worker-75) [52dee25c] Setting new tasks map. The map contains now 0 tasks >2014-07-30 10:46:29,061 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (DefaultQuartzScheduler_Worker-75) [52dee25c] Cleared all tasks of pool 00000002-0002-0002-0002-0000000001d9. >2014-07-30 10:47:02,486 INFO [org.ovirt.engine.core.bll.RunVmOnceCommand] (ajp--127.0.0.1-8702-4) [31cbf1b] Lock Acquired to object EngineLock [exclusiveLocks= key: 9042bb21-3f27-48bb-ab26-c0d8b18c840d value: VM >, sharedLocks= ] >2014-07-30 10:47:02,612 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (ajp--127.0.0.1-8702-4) START, IsVmDuringInitiatingVDSCommand( vmId = 9042bb21-3f27-48bb-ab26-c0d8b18c840d), log id: 7a3461a9 >2014-07-30 10:47:02,614 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (ajp--127.0.0.1-8702-4) FINISH, IsVmDuringInitiatingVDSCommand, return: false, log id: 7a3461a9 >2014-07-30 10:47:02,655 INFO [org.ovirt.engine.core.bll.RunVmOnceCommand] (ajp--127.0.0.1-8702-4) Running command: RunVmOnceCommand internal: false. Entities affected : ID: 9042bb21-3f27-48bb-ab26-c0d8b18c840d Type: VM >2014-07-30 10:47:02,683 INFO [org.ovirt.engine.core.bll.RunVmCommand] (ajp--127.0.0.1-8702-4) Running VM with attached cd ovirt-node-iso-3.1.0-0.999.999.20140317175137gitba60c851296.el6.iso >2014-07-30 10:47:02,686 INFO [org.ovirt.engine.core.vdsbroker.UpdateVmDynamicDataVDSCommand] (ajp--127.0.0.1-8702-4) START, UpdateVmDynamicDataVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vmDynamic=org.ovirt.engine.core.common.businessentities.VmDynamic@66810750), log id: 6401d514 >2014-07-30 10:47:02,690 INFO [org.ovirt.engine.core.vdsbroker.UpdateVmDynamicDataVDSCommand] (ajp--127.0.0.1-8702-4) FINISH, UpdateVmDynamicDataVDSCommand, log id: 6401d514 >2014-07-30 10:47:02,694 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.IsoPrefixVDSCommand] (ajp--127.0.0.1-8702-4) START, IsoPrefixVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId=00000002-0002-0002-0002-0000000001d9), log id: 49805003 >2014-07-30 10:47:02,714 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.IsoPrefixVDSCommand] (ajp--127.0.0.1-8702-4) FINISH, IsoPrefixVDSCommand, return: /rhev/data-center/mnt/10.34.63.204:_home_iso_shared/0c78b4d6-ba00-4d3e-9f9f-65c7d5899d71/images/11111111-1111-1111-1111-111111111111, log id: 49805003 >2014-07-30 10:47:02,728 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (ajp--127.0.0.1-8702-4) START, CreateVmVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vmId=9042bb21-3f27-48bb-ab26-c0d8b18c840d, vm=VM [ovirtnode]), log id: 761cd6f4 >2014-07-30 10:47:02,737 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (ajp--127.0.0.1-8702-4) START, CreateVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vmId=9042bb21-3f27-48bb-ab26-c0d8b18c840d, vm=VM [ovirtnode]), log id: 8bea027 >2014-07-30 10:47:02,776 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (ajp--127.0.0.1-8702-4) org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand kvmEnable=true,keyboardLayout=en-us,nice=0,displayNetwork=ovirtmgmt,copyPasteEnable=true,timeOffset=0,transparentHugePages=true,vmId=9042bb21-3f27-48bb-ab26-c0d8b18c840d,acpiEnable=true,custom={},spiceSslCipherSuite=DEFAULT,memSize=1024,smp=1,emulatedMachine=rhel6.5.0,vmType=kvm,memGuaranteedSize=1024,display=qxl,smartcardEnable=false,bootMenuEnable=false,spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,susbredir,ssmartcard,smpCoresPerSocket=1,numaTune={mode=preferred},maxVCpus=160,devices=[{address={bus=0x00, domain=0x0000, slot=0x02, type=pci, function=0x0}, specParams={vram=32768, heads=1}, device=qxl, type=video, deviceId=05cedad9-c992-4b63-89e9-3708ee7e3d89}, {shared=false, bootOrder=2, iface=ide, index=2, specParams={}, path=/rhev/data-center/mnt/10.34.63.204:_home_iso_shared/0c78b4d6-ba00-4d3e-9f9f-65c7d5899d71/images/11111111-1111-1111-1111-111111111111/ovirt-node-iso-3.1.0-0.999.999.20140317175137gitba60c851296.el6.iso, device=cdrom, type=disk, readonly=true, deviceId=4329cc23-372a-4399-a974-a87ca8cd3d0d}, {shared=false, index=0, volumeID=1efc6019-96fd-4b59-8289-6ce7fe14a614, propagateErrors=off, format=raw, type=disk, bootOrder=1, iface=virtio, address={bus=0x00, domain=0x0000, slot=0x06, type=pci, function=0x0}, domainID=71f82fcd-4fa6-4f56-b228-f4fa4bd4f752, imageID=cc7b5e15-0557-460a-af76-7daa89c27d9f, specParams={}, optional=false, device=disk, poolID=00000002-0002-0002-0002-0000000001d9, readonly=false, deviceId=cc7b5e15-0557-460a-af76-7daa89c27d9f}, {bootOrder=3, nicModel=pv, address={bus=0x00, domain=0x0000, slot=0x03, type=pci, function=0x0}, specParams={outbound={}, inbound={}}, macAddr=00:1a:4a:ae:a8:01, device=bridge, linkActive=true, type=interface, filter=vdsm-no-mac-spoofing, network=ovirtmgmt, deviceId=17e210d5-07a0-4c2a-b63d-245a4f01ec21}, {index=0, model=virtio-scsi, address={bus=0x00, domain=0x0000, slot=0x04, type=pci, function=0x0}, specParams={}, device=scsi, type=controller, deviceId=77986732-043a-4105-9c7b-231a79a0db6f}],vmName=ovirtnode,cpuType=Westmere,fileTransferEnable=true >2014-07-30 10:47:02,805 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (ajp--127.0.0.1-8702-4) FINISH, CreateVDSCommand, log id: 8bea027 >2014-07-30 10:47:02,812 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (ajp--127.0.0.1-8702-4) FINISH, CreateVmVDSCommand, return: WaitForLaunch, log id: 761cd6f4 >2014-07-30 10:47:02,813 INFO [org.ovirt.engine.core.bll.RunVmOnceCommand] (ajp--127.0.0.1-8702-4) Lock freed to object EngineLock [exclusiveLocks= key: 9042bb21-3f27-48bb-ab26-c0d8b18c840d value: VM >, sharedLocks= ] >2014-07-30 10:47:02,824 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-4) Correlation ID: 31cbf1b, Job ID: 4a7ed238-134e-41ad-98e9-6748c617f590, Call Stack: null, Custom Event ID: -1, Message: VM ovirtnode was started by admin (Host: rack). >2014-07-30 10:47:03,941 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-62) [2e742a17] VM ovirtnode 9042bb21-3f27-48bb-ab26-c0d8b18c840d moved from WaitForLaunch --> PoweringUp >2014-07-30 10:47:03,942 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVdsCommand] (DefaultQuartzScheduler_Worker-62) [2e742a17] START, FullListVdsCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b], vmIds=[9042bb21-3f27-48bb-ab26-c0d8b18c840d]), log id: 31d148f5 >2014-07-30 10:47:03,950 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVdsCommand] (DefaultQuartzScheduler_Worker-62) [2e742a17] FINISH, FullListVdsCommand, return: [{displaySecurePort=5901, kvmEnable=true, nicModel=rtl8139,pv, keyboardLayout=en-us, displayIp=10.34.62.208, pauseCode=NOERR, nice=0, displayNetwork=ovirtmgmt, copyPasteEnable=true, timeOffset=0, transparentHugePages=true, vmId=9042bb21-3f27-48bb-ab26-c0d8b18c840d, acpiEnable=true, custom={}, spiceSslCipherSuite=DEFAULT, memSize=1024, smp=1, displayPort=5900, emulatedMachine=rhel6.5.0, vmType=kvm, status=Up, memGuaranteedSize=1024, display=qxl, pid=23146, smartcardEnable=false, bootMenuEnable=false, spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,susbredir,ssmartcard, numaTune={mode=preferred}, smpCoresPerSocket=1, maxVCpus=160, clientIp=, devices=[Ljava.lang.Object;@230ce660, vmName=ovirtnode, fileTransferEnable=true, cpuType=Westmere}], log id: 31d148f5 >2014-07-30 10:47:03,957 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-62) [2e742a17] Received a memballoon Device without an address when processing VM 9042bb21-3f27-48bb-ab26-c0d8b18c840d devices, skipping device: {alias=balloon0, specParams={model=none}, device=memballoon, type=balloon} >2014-07-30 10:47:03,959 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-62) [2e742a17] Received a spice Device without an address when processing VM 9042bb21-3f27-48bb-ab26-c0d8b18c840d devices, skipping device: {port=5900, specParams={spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,susbredir,ssmartcard, keyMap=en-us, displayNetwork=ovirtmgmt, copyPasteEnable=true, displayIp=10.34.62.208}, device=spice, tlsPort=5901, type=graphics} >2014-07-30 10:47:17,404 INFO [org.ovirt.engine.core.bll.SetVmTicketCommand] (ajp--127.0.0.1-8702-6) [ff43ced] Running command: SetVmTicketCommand internal: false. Entities affected : ID: 9042bb21-3f27-48bb-ab26-c0d8b18c840d Type: VM >2014-07-30 10:47:17,409 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetVmTicketVDSCommand] (ajp--127.0.0.1-8702-6) [ff43ced] START, SetVmTicketVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vmId=9042bb21-3f27-48bb-ab26-c0d8b18c840d, ticket=LQf3rGZKSh2M, validTime=120,m userName=admin, userId=fdfc627c-d875-11e0-90f0-83df133b58cc), log id: 160a92aa >2014-07-30 10:47:17,443 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetVmTicketVDSCommand] (ajp--127.0.0.1-8702-6) [ff43ced] FINISH, SetVmTicketVDSCommand, log id: 160a92aa >2014-07-30 10:47:17,453 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) [ff43ced] Correlation ID: ff43ced, Call Stack: null, Custom Event ID: -1, Message: user admin initiated console session for VM ovirtnode >2014-07-30 10:47:22,211 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-50) [375fcc6] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin is connected to VM ovirtnode. >2014-07-30 10:48:04,665 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-32) VM ovirtnode 9042bb21-3f27-48bb-ab26-c0d8b18c840d moved from PoweringUp --> Up >2014-07-30 10:48:04,678 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-32) Correlation ID: 31cbf1b, Job ID: 4a7ed238-134e-41ad-98e9-6748c617f590, Call Stack: null, Custom Event ID: -1, Message: VM ovirtnode started on Host rack >2014-07-30 10:50:43,081 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-100) [12a64afb] Failed in SpmStatusVDS method >2014-07-30 10:50:43,083 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-100) [12a64afb] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to SpmStatusVDS, error = (-202, 'Sanlock resource read failure', 'Sanlock exception'), code = 100 >2014-07-30 10:50:43,088 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-100) [52a75f18] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-07-30 10:50:43,092 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-100) [52a75f18] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Active to Unknown, reason : null >2014-07-30 10:50:43,094 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-100) [52a75f18] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 0c78b4d6-ba00-4d3e-9f9f-65c7d5899d71 status from Active to Unknown, reason : null >2014-07-30 10:50:43,106 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-100) [52a75f18] Correlation ID: 52a75f18, Call Stack: null, Custom Event ID: -1, Message: Invalid status on Data Center Default. Setting Data Center status to Non Responsive (On host rack, Error: General Exception). >2014-07-30 10:50:45,050 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-36) domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 in problem. vds: rack >2014-07-30 10:50:49,121 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-100) [52a75f18] hostFromVds::selectedVds - rack, spmStatus SPM, storage pool Default >2014-07-30 10:50:49,127 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-100) [52a75f18] Initialize Irs proxy from vds: 10.34.62.208 >2014-07-30 10:50:49,134 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-100) [52a75f18] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208). >2014-07-30 10:50:49,136 WARN [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-8-thread-34) Executing a command: java.util.concurrent.FutureTask , but note that there are 1 tasks in the queue. >2014-07-30 10:50:49,162 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-34) [52a75f18] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 638a5d14 >2014-07-30 10:50:51,715 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-34) [52a75f18] -- executeIrsBrokerCommand: Attempting on storage pool 00000002-0002-0002-0002-0000000001d9 >2014-07-30 10:50:51,717 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-34) [52a75f18] START, HSMGetAllTasksInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 45200898 >2014-07-30 10:50:51,722 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-34) [52a75f18] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 45200898 >2014-07-30 10:50:51,723 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-34) [52a75f18] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 638a5d14 >2014-07-30 10:50:51,724 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (org.ovirt.thread.pool-8-thread-34) [52a75f18] Discovered no tasks on Storage Pool Default >2014-07-30 10:51:00,169 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-27) Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 recovered from problem. vds: rack >2014-07-30 10:51:00,170 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-27) Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 has recovered from problem. No active host in the DC is reporting it as problematic, so clearing the domain recovery timer. >2014-07-30 10:52:30,878 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-60) VM ovirtnode 9042bb21-3f27-48bb-ab26-c0d8b18c840d moved from Up --> RebootInProgress >2014-07-30 10:52:30,881 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVdsCommand] (DefaultQuartzScheduler_Worker-60) START, FullListVdsCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b], vmIds=[9042bb21-3f27-48bb-ab26-c0d8b18c840d]), log id: 5b6e6aa3 >2014-07-30 10:52:30,888 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVdsCommand] (DefaultQuartzScheduler_Worker-60) FINISH, FullListVdsCommand, return: [{displaySecurePort=5901, kvmEnable=true, nicModel=rtl8139,pv, keyboardLayout=en-us, displayIp=10.34.62.208, pauseCode=NOERR, nice=0, displayNetwork=ovirtmgmt, copyPasteEnable=true, timeOffset=-1, transparentHugePages=true, vmId=9042bb21-3f27-48bb-ab26-c0d8b18c840d, acpiEnable=true, custom={}, spiceSslCipherSuite=DEFAULT, memSize=1024, smp=1, displayPort=5900, emulatedMachine=rhel6.5.0, vmType=kvm, status=Up, memGuaranteedSize=1024, display=qxl, pid=23146, smartcardEnable=false, bootMenuEnable=false, spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,susbredir,ssmartcard, numaTune={mode=preferred}, smpCoresPerSocket=1, maxVCpus=160, clientIp=10.34.130.128, devices=[Ljava.lang.Object;@2e1b223a, vmName=ovirtnode, fileTransferEnable=true, cpuType=Westmere}], log id: 5b6e6aa3 >2014-07-30 10:52:30,894 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-60) Received a memballoon Device without an address when processing VM 9042bb21-3f27-48bb-ab26-c0d8b18c840d devices, skipping device: {alias=balloon0, specParams={model=none}, device=memballoon, type=balloon} >2014-07-30 10:52:30,896 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-60) Received a spice Device without an address when processing VM 9042bb21-3f27-48bb-ab26-c0d8b18c840d devices, skipping device: {port=5900, specParams={spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,susbredir,ssmartcard, keyMap=en-us, displayNetwork=ovirtmgmt, copyPasteEnable=true, displayIp=10.34.62.208}, device=spice, tlsPort=5901, type=graphics} >2014-07-30 10:53:19,480 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-32) VM ovirtnode 9042bb21-3f27-48bb-ab26-c0d8b18c840d moved from RebootInProgress --> Up >2014-07-30 10:58:49,226 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-24) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin got disconnected from VM ovirtnode. >2014-07-30 10:58:52,573 INFO [org.ovirt.engine.core.bll.StopVmCommand] (org.ovirt.thread.pool-8-thread-30) [536a6748] Running command: StopVmCommand internal: false. Entities affected : ID: 9042bb21-3f27-48bb-ab26-c0d8b18c840d Type: VM >2014-07-30 10:58:52,578 INFO [org.ovirt.engine.core.vdsbroker.DestroyVmVDSCommand] (org.ovirt.thread.pool-8-thread-30) [536a6748] START, DestroyVmVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vmId=9042bb21-3f27-48bb-ab26-c0d8b18c840d, force=false, secondsToWait=0, gracefully=false, reason=), log id: 21fcde55 >2014-07-30 10:58:52,584 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (org.ovirt.thread.pool-8-thread-30) [536a6748] START, DestroyVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vmId=9042bb21-3f27-48bb-ab26-c0d8b18c840d, force=false, secondsToWait=0, gracefully=false, reason=), log id: 3f7d478f >2014-07-30 10:58:53,466 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (org.ovirt.thread.pool-8-thread-30) [536a6748] FINISH, DestroyVDSCommand, log id: 3f7d478f >2014-07-30 10:58:53,472 INFO [org.ovirt.engine.core.vdsbroker.DestroyVmVDSCommand] (org.ovirt.thread.pool-8-thread-30) [536a6748] FINISH, DestroyVmVDSCommand, return: Down, log id: 21fcde55 >2014-07-30 10:58:53,483 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-30) [536a6748] Correlation ID: 536a6748, Job ID: fa6da149-a69e-4562-b193-07632374556a, Call Stack: null, Custom Event ID: -1, Message: VM ovirtnode powered off by admin (Host: rack) (Reason: Not Specified). >2014-07-30 10:58:53,530 INFO [org.ovirt.engine.core.bll.ProcessDownVmCommand] (org.ovirt.thread.pool-8-thread-20) [13562194] Running command: ProcessDownVmCommand internal: true. >2014-07-30 11:00:35,989 INFO [org.ovirt.engine.core.bll.RunVmCommand] (ajp--127.0.0.1-8702-4) [a757eeb] Lock Acquired to object EngineLock [exclusiveLocks= key: 9042bb21-3f27-48bb-ab26-c0d8b18c840d value: VM >, sharedLocks= ] >2014-07-30 11:00:36,070 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (ajp--127.0.0.1-8702-4) [a757eeb] START, IsVmDuringInitiatingVDSCommand( vmId = 9042bb21-3f27-48bb-ab26-c0d8b18c840d), log id: 5e2cf264 >2014-07-30 11:00:36,071 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (ajp--127.0.0.1-8702-4) [a757eeb] FINISH, IsVmDuringInitiatingVDSCommand, return: false, log id: 5e2cf264 >2014-07-30 11:00:36,109 INFO [org.ovirt.engine.core.bll.RunVmCommand] (org.ovirt.thread.pool-8-thread-16) [a757eeb] Running command: RunVmCommand internal: false. Entities affected : ID: 9042bb21-3f27-48bb-ab26-c0d8b18c840d Type: VM >2014-07-30 11:00:36,132 INFO [org.ovirt.engine.core.vdsbroker.UpdateVmDynamicDataVDSCommand] (org.ovirt.thread.pool-8-thread-16) [a757eeb] START, UpdateVmDynamicDataVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vmDynamic=org.ovirt.engine.core.common.businessentities.VmDynamic@823b1cc3), log id: 2578f29c >2014-07-30 11:00:36,136 INFO [org.ovirt.engine.core.vdsbroker.UpdateVmDynamicDataVDSCommand] (org.ovirt.thread.pool-8-thread-16) [a757eeb] FINISH, UpdateVmDynamicDataVDSCommand, log id: 2578f29c >2014-07-30 11:00:36,141 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (org.ovirt.thread.pool-8-thread-16) [a757eeb] START, CreateVmVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vmId=9042bb21-3f27-48bb-ab26-c0d8b18c840d, vm=VM [ovirtnode]), log id: e68d50e >2014-07-30 11:00:36,145 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (org.ovirt.thread.pool-8-thread-16) [a757eeb] START, CreateVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vmId=9042bb21-3f27-48bb-ab26-c0d8b18c840d, vm=VM [ovirtnode]), log id: 54c3a16 >2014-07-30 11:00:36,165 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (org.ovirt.thread.pool-8-thread-16) [a757eeb] org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand kvmEnable=true,keyboardLayout=en-us,nice=0,displayNetwork=ovirtmgmt,copyPasteEnable=true,timeOffset=0,transparentHugePages=true,vmId=9042bb21-3f27-48bb-ab26-c0d8b18c840d,acpiEnable=true,custom={},spiceSslCipherSuite=DEFAULT,memSize=1024,smp=1,emulatedMachine=rhel6.5.0,vmType=kvm,memGuaranteedSize=1024,display=qxl,smartcardEnable=false,bootMenuEnable=false,spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,susbredir,ssmartcard,smpCoresPerSocket=1,numaTune={mode=preferred},maxVCpus=160,devices=[{address={bus=0x00, domain=0x0000, slot=0x02, type=pci, function=0x0}, specParams={vram=32768, heads=1}, device=qxl, type=video, deviceId=05cedad9-c992-4b63-89e9-3708ee7e3d89}, {shared=false, iface=ide, index=2, specParams={path=}, path=, device=cdrom, type=disk, readonly=true, deviceId=cbd5ab2a-dc25-451c-9407-ab762c3570da}, {shared=false, index=0, volumeID=1efc6019-96fd-4b59-8289-6ce7fe14a614, propagateErrors=off, format=raw, type=disk, iface=virtio, address={bus=0x00, domain=0x0000, slot=0x06, type=pci, function=0x0}, domainID=71f82fcd-4fa6-4f56-b228-f4fa4bd4f752, imageID=cc7b5e15-0557-460a-af76-7daa89c27d9f, specParams={}, optional=false, device=disk, poolID=00000002-0002-0002-0002-0000000001d9, readonly=false, deviceId=cc7b5e15-0557-460a-af76-7daa89c27d9f}, {nicModel=pv, address={bus=0x00, domain=0x0000, slot=0x03, type=pci, function=0x0}, specParams={outbound={}, inbound={}}, macAddr=00:1a:4a:ae:a8:01, device=bridge, linkActive=true, type=interface, filter=vdsm-no-mac-spoofing, network=ovirtmgmt, deviceId=17e210d5-07a0-4c2a-b63d-245a4f01ec21}, {index=0, model=virtio-scsi, address={bus=0x00, domain=0x0000, slot=0x04, type=pci, function=0x0}, specParams={}, device=scsi, type=controller, deviceId=77986732-043a-4105-9c7b-231a79a0db6f}],vmName=ovirtnode,cpuType=Westmere,fileTransferEnable=true >2014-07-30 11:00:36,196 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (org.ovirt.thread.pool-8-thread-16) [a757eeb] FINISH, CreateVDSCommand, log id: 54c3a16 >2014-07-30 11:00:36,202 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (org.ovirt.thread.pool-8-thread-16) [a757eeb] FINISH, CreateVmVDSCommand, return: WaitForLaunch, log id: e68d50e >2014-07-30 11:00:36,203 INFO [org.ovirt.engine.core.bll.RunVmCommand] (org.ovirt.thread.pool-8-thread-16) [a757eeb] Lock freed to object EngineLock [exclusiveLocks= key: 9042bb21-3f27-48bb-ab26-c0d8b18c840d value: VM >, sharedLocks= ] >2014-07-30 11:00:36,209 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-16) [a757eeb] Correlation ID: a757eeb, Job ID: dcca760f-4b55-4584-bc7d-fb073e231cd0, Call Stack: null, Custom Event ID: -1, Message: VM ovirtnode was started by admin (Host: rack). >2014-07-30 11:00:38,351 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-89) VM ovirtnode 9042bb21-3f27-48bb-ab26-c0d8b18c840d moved from WaitForLaunch --> PoweringUp >2014-07-30 11:00:38,352 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVdsCommand] (DefaultQuartzScheduler_Worker-89) START, FullListVdsCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b], vmIds=[9042bb21-3f27-48bb-ab26-c0d8b18c840d]), log id: 580a0f2d >2014-07-30 11:00:38,368 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVdsCommand] (DefaultQuartzScheduler_Worker-89) FINISH, FullListVdsCommand, return: [{displaySecurePort=5901, kvmEnable=true, nicModel=rtl8139,pv, keyboardLayout=en-us, displayIp=10.34.62.208, pauseCode=NOERR, nice=0, displayNetwork=ovirtmgmt, copyPasteEnable=true, timeOffset=0, transparentHugePages=true, vmId=9042bb21-3f27-48bb-ab26-c0d8b18c840d, acpiEnable=true, custom={}, spiceSslCipherSuite=DEFAULT, memSize=1024, smp=1, displayPort=5900, emulatedMachine=rhel6.5.0, vmType=kvm, status=Up, memGuaranteedSize=1024, display=qxl, pid=24451, smartcardEnable=false, bootMenuEnable=false, spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,susbredir,ssmartcard, numaTune={mode=preferred}, smpCoresPerSocket=1, maxVCpus=160, clientIp=, devices=[Ljava.lang.Object;@7c6a8082, vmName=ovirtnode, fileTransferEnable=true, cpuType=Westmere}], log id: 580a0f2d >2014-07-30 11:00:38,374 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-89) Received a memballoon Device without an address when processing VM 9042bb21-3f27-48bb-ab26-c0d8b18c840d devices, skipping device: {alias=balloon0, specParams={model=none}, device=memballoon, type=balloon} >2014-07-30 11:00:38,375 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-89) Received a spice Device without an address when processing VM 9042bb21-3f27-48bb-ab26-c0d8b18c840d devices, skipping device: {port=5900, specParams={spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,susbredir,ssmartcard, keyMap=en-us, displayNetwork=ovirtmgmt, copyPasteEnable=true, displayIp=10.34.62.208}, device=spice, tlsPort=5901, type=graphics} >2014-07-30 11:01:39,046 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-18) VM ovirtnode 9042bb21-3f27-48bb-ab26-c0d8b18c840d moved from PoweringUp --> Up >2014-07-30 11:01:39,060 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-18) Correlation ID: a757eeb, Job ID: dcca760f-4b55-4584-bc7d-fb073e231cd0, Call Stack: null, Custom Event ID: -1, Message: VM ovirtnode started on Host rack >2014-07-30 11:10:10,427 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-43) domain 0c78b4d6-ba00-4d3e-9f9f-65c7d5899d71:str01-iso-shared in problem. vds: rack >2014-07-30 11:10:25,553 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-25) domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 in problem. vds: rack >2014-07-30 11:13:42,106 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-47) VM ovirtnode 9042bb21-3f27-48bb-ab26-c0d8b18c840d moved from Up --> NotResponding >2014-07-30 11:13:42,167 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-47) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: VM ovirtnode is not responding. >2014-07-30 11:13:45,220 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-73) VM ovirtnode 9042bb21-3f27-48bb-ab26-c0d8b18c840d moved from NotResponding --> Up >2014-07-30 11:15:10,430 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-39) starting processDomainRecovery for domain 0c78b4d6-ba00-4d3e-9f9f-65c7d5899d71:str01-iso-shared >2014-07-30 11:15:10,485 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-39) Domain 0c78b4d6-ba00-4d3e-9f9f-65c7d5899d71:str01-iso-shared was reported by all hosts in status UP as problematic. Moving the domain to NonOperational. >2014-07-30 11:15:10,508 INFO [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-39) [d7a3c40] Lock Acquired to object EngineLock [exclusiveLocks= key: 0c78b4d6-ba00-4d3e-9f9f-65c7d5899d71 value: STORAGE >, sharedLocks= ] >2014-07-30 11:15:10,515 INFO [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-39) [d7a3c40] Running command: DeactivateStorageDomainCommand internal: true. Entities affected : ID: 0c78b4d6-ba00-4d3e-9f9f-65c7d5899d71 Type: Storage >2014-07-30 11:15:10,524 INFO [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-39) [d7a3c40] Lock freed to object EngineLock [exclusiveLocks= key: 0c78b4d6-ba00-4d3e-9f9f-65c7d5899d71 value: STORAGE >, sharedLocks= ] >2014-07-30 11:15:10,542 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-39) [d7a3c40] Correlation ID: d7a3c40, Job ID: 4781ad59-ef44-4d90-b7a2-63660226cd8f, Call Stack: null, Custom Event ID: -1, Message: Storage Domain str01-iso-shared (Data Center Default) was deactivated by system because it's not visible by any of the hosts. >2014-07-30 11:15:10,548 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-39) [d7a3c40] Removing vds [] from the domain in maintenance cache >2014-07-30 11:15:10,549 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-39) [d7a3c40] Removing host(s) [] from hosts unseen domain report cache >2014-07-30 11:15:20,790 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-69) Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2014-07-30 11:15:20,795 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-69) [52d379dc] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-07-30 11:15:20,798 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-69) [52d379dc] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Active to Unknown, reason : null >2014-07-30 11:15:20,807 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-69) [52d379dc] Correlation ID: 52d379dc, Call Stack: null, Custom Event ID: -1, Message: Invalid status on Data Center Default. Setting Data Center status to Non Responsive (On host rack, Error: Cannot access master storage domain). >2014-07-30 11:15:25,556 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-47) starting processDomainRecovery for domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 >2014-07-30 11:15:25,561 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-47) Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 was reported by all hosts in status UP as problematic. Not moving the domain to NonOperational because it is being reconstructed now. >2014-07-30 11:15:25,591 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-47) [7aeae3d7] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage >2014-07-30 11:15:25,621 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-47) [7aeae3d7] START, ResetIrsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, ignoreStopFailed = true), log id: 3b907c7a >2014-07-30 11:15:25,628 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-47) [7aeae3d7] START, SpmStopVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9), log id: 4d1ddc64 >2014-07-30 11:15:25,635 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-47) [7aeae3d7] SpmStopVDSCommand::Stopping SPM on vds rack, pool id 00000002-0002-0002-0002-0000000001d9 >2014-07-30 11:18:20,836 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-69) [52d379dc] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2014-07-30 11:18:20,838 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-69) [52d379dc] hostFromVds::selectedVds - rack, spmStatus returned null! >2014-07-30 11:18:20,839 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-69) [52d379dc] IrsBroker::Failed::GetStoragePoolInfoVDS >2014-07-30 11:18:20,840 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-69) [52d379dc] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2014-07-30 11:18:25,637 WARN [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-47) [7aeae3d7] could not stop spm of pool 00000002-0002-0002-0002-0000000001d9 on vds 17f7fe56-5f32-40ef-b098-611b84484e0b - reason: org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcRunTimeException: java.util.concurrent.TimeoutException >2014-07-30 11:18:25,641 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-47) [7aeae3d7] FINISH, SpmStopVDSCommand, log id: 4d1ddc64 >2014-07-30 11:18:25,649 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-47) [7aeae3d7] FINISH, ResetIrsVDSCommand, log id: 3b907c7a >2014-07-30 11:18:25,662 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-47) [7aeae3d7] Correlation ID: 7aeae3d7, Job ID: 371e7693-0331-4087-9db8-836d853059d5, Call Stack: null, Custom Event ID: -1, Message: Failed to Reconstruct Master Domain for Data Center Default. >2014-07-30 11:18:25,669 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-47) [7aeae3d7] Removing vds [] from the domain in maintenance cache >2014-07-30 11:18:25,670 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-47) [7aeae3d7] Removing host(s) [] from hosts unseen domain report cache >2014-07-30 11:18:25,671 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-33) [2e41028] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue >2014-07-30 11:18:25,672 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-33) [2e41028] The following operation Event [eventType=RECONSTRUCT, storagePoolId=00000002-0002-0002-0002-0000000001d9, domainId=71f82fcd-4fa6-4f56-b228-f4fa4bd4f752, vdsId=null, description=IrsBrokerCommand.startReconstruct()] was cancelled, because of reconstruct was run before >2014-07-30 11:18:25,674 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-33) [2e41028] The following operation Event [eventType=DOMAINMONITORING, storagePoolId=00000002-0002-0002-0002-0000000001d9, domainId=null, vdsId=17f7fe56-5f32-40ef-b098-611b84484e0b, description=] was cancelled, because of reconstruct was run before >2014-07-30 11:19:20,800 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-30) [dd7f8b1] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: VDSNetworkException: org.apache.commons.httpclient.NoHttpResponseException: The server 10.34.62.208 failed to respond >2014-07-30 11:19:20,804 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-30) [dd7f8b1] hostFromVds::selectedVds - rack, spmStatus returned null! >2014-07-30 11:19:23,257 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-39) [4a2f7e33] VM ovirtnode 9042bb21-3f27-48bb-ab26-c0d8b18c840d moved from Up --> Paused >2014-07-30 11:19:30,835 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-15) hostFromVds::selectedVds - rack, spmStatus Unknown_Pool, storage pool Default >2014-07-30 11:19:30,841 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-15) START, ConnectStoragePoolVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, masterVersion = 1), log id: 446581d7 >2014-07-30 11:19:41,500 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-97) Storage Domain 0c78b4d6-ba00-4d3e-9f9f-65c7d5899d71:str01-iso-shared was reported by Host rack as Active in Pool 00000002-0002-0002-0002-0000000001d9, moving to active status >2014-07-30 11:19:41,506 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-97) Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 was reported by Host rack as Active in Pool 00000002-0002-0002-0002-0000000001d9, moving to active status >2014-07-30 11:20:31,315 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-15) FINISH, ConnectStoragePoolVDSCommand, log id: 446581d7 >2014-07-30 11:20:31,414 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-15) hostFromVds::Connected host to pool - selectedVds - rack, spmStatus Free, storage pool Default >2014-07-30 11:20:31,420 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-15) starting spm on vds rack, storage pool Default, prevId -1, LVER -1 >2014-07-30 11:20:31,423 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-15) START, SpmStartVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, prevId=-1, prevLVER=-1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 237983e6 >2014-07-30 11:20:31,472 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-15) spmStart polling started: taskId = 7a33a35d-a2c5-48ad-b7f5-907a5bf36dc4 >2014-07-30 11:20:32,478 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-15) spmStart polling ended: taskId = 7a33a35d-a2c5-48ad-b7f5-907a5bf36dc4 task status = finished >2014-07-30 11:20:32,537 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-15) spmStart polling ended, spm status: SPM >2014-07-30 11:20:32,539 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-15) START, HSMClearTaskVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, taskId=7a33a35d-a2c5-48ad-b7f5-907a5bf36dc4), log id: 510712fd >2014-07-30 11:20:32,585 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-15) FINISH, HSMClearTaskVDSCommand, log id: 510712fd >2014-07-30 11:20:32,585 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-15) FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@417c9d1, log id: 237983e6 >2014-07-30 11:20:32,589 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-15) Initialize Irs proxy from vds: 10.34.62.208 >2014-07-30 11:20:32,596 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-15) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208). >2014-07-30 11:20:32,598 WARN [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-8-thread-21) Executing a command: java.util.concurrent.FutureTask , but note that there are 1 tasks in the queue. >2014-07-30 11:20:32,616 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-21) START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 71ab0f30 >2014-07-30 11:20:32,633 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-21) -- executeIrsBrokerCommand: Attempting on storage pool 00000002-0002-0002-0002-0000000001d9 >2014-07-30 11:20:32,635 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-21) START, HSMGetAllTasksInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 3c5a4d50 >2014-07-30 11:20:32,641 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-21) FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 3c5a4d50 >2014-07-30 11:20:32,642 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-21) FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 71ab0f30 >2014-07-30 11:20:32,642 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (org.ovirt.thread.pool-8-thread-21) Discovered no tasks on Storage Pool Default >2014-07-30 11:23:55,117 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-40) Principal admin::internal synchronized >2014-07-30 11:24:01,527 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-46) Attempting to update VMs/Templates Ovf. >2014-07-30 11:24:01,673 INFO [org.ovirt.engine.core.bll.ProcessOvfUpdateForStorageDomainCommand] (DefaultQuartzScheduler_Worker-46) [7a2d8c73] Lock Acquired to object EngineLock [exclusiveLocks= key: 07c061c2-77f4-4c02-8402-e4dff773b216 value: DISK >key: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 value: STORAGE >key: 90108fe1-56a4-4f0e-b301-fa0ef46ba717 value: DISK >, sharedLocks= key: 00000002-0002-0002-0002-0000000001d9 value: OVF_UPDATE >] >2014-07-30 11:24:01,682 INFO [org.ovirt.engine.core.bll.ProcessOvfUpdateForStorageDomainCommand] (DefaultQuartzScheduler_Worker-46) [7a2d8c73] Running command: ProcessOvfUpdateForStorageDomainCommand internal: true. Entities affected : ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage >2014-07-30 11:24:01,706 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (DefaultQuartzScheduler_Worker-46) [7a2d8c73] START, SetVolumeDescriptionVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, storageDomainId = 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752, imageGroupId = 90108fe1-56a4-4f0e-b301-fa0ef46ba717, imageId = 8596469e-37df-4d9e-9793-d238238131f5), log id: 56c3f1a1 >2014-07-30 11:24:01,711 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (DefaultQuartzScheduler_Worker-46) [7a2d8c73] -- executeIrsBrokerCommand: calling 'setVolumeDescription' >2014-07-30 11:24:01,714 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (DefaultQuartzScheduler_Worker-46) [7a2d8c73] -- setVolumeDescription parameters: > spUUID=00000002-0002-0002-0002-0000000001d9 > sdUUID=71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 > imageGroupGUID=90108fe1-56a4-4f0e-b301-fa0ef46ba717 > volUUID=8596469e-37df-4d9e-9793-d238238131f5 > description={"Updated":false,"Disk Description":"OVF_STORE","Storage Domains":[{"uuid":"71f82fcd-4fa6-4f56-b228-f4fa4bd4f752"}],"Last Updated":"Fri Jul 25 14:23:55 CEST 2014"} > >2014-07-30 11:24:04,517 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (DefaultQuartzScheduler_Worker-46) [7a2d8c73] FINISH, SetVolumeDescriptionVDSCommand, log id: 56c3f1a1 >2014-07-30 11:24:04,550 INFO [org.ovirt.engine.core.bll.UploadStreamCommand] (DefaultQuartzScheduler_Worker-46) [233b68fb] Running command: UploadStreamCommand internal: true. Entities affected : ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage >2014-07-30 11:24:04,552 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.UploadStreamVDSCommand] (DefaultQuartzScheduler_Worker-46) [233b68fb] START, UploadStreamVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 7268035d >2014-07-30 11:24:04,553 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.UploadStreamVDSCommand] (DefaultQuartzScheduler_Worker-46) [233b68fb] -- executeVdsBrokerCommand: >2014-07-30 11:24:04,554 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.UploadStreamVDSCommand] (DefaultQuartzScheduler_Worker-46) [233b68fb] -- parameters: > spUUID=00000002-0002-0002-0002-0000000001d9 > sdUUID=71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 > imageGUID=90108fe1-56a4-4f0e-b301-fa0ef46ba717 > volUUID=8596469e-37df-4d9e-9793-d238238131f5 > size=20,480 > >2014-07-30 11:24:20,251 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.UploadStreamVDSCommand] (DefaultQuartzScheduler_Worker-46) [233b68fb] FINISH, UploadStreamVDSCommand, log id: 7268035d >2014-07-30 11:24:20,259 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (DefaultQuartzScheduler_Worker-46) [233b68fb] CommandAsyncTask::Adding CommandMultiAsyncTasks object for command 03335b91-a86c-46c6-8274-c21b3b5a9179 >2014-07-30 11:24:20,261 INFO [org.ovirt.engine.core.bll.CommandMultiAsyncTasks] (DefaultQuartzScheduler_Worker-46) [233b68fb] CommandMultiAsyncTasks::AttachTask: Attaching task 13cca31c-d228-4247-a3cd-c324ae05433e to command 03335b91-a86c-46c6-8274-c21b3b5a9179. >2014-07-30 11:24:20,274 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (DefaultQuartzScheduler_Worker-46) [233b68fb] Adding task 13cca31c-d228-4247-a3cd-c324ae05433e (Parent Command ProcessOvfUpdateForStorageDomain, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters), polling hasn't started yet.. >2014-07-30 11:24:20,277 INFO [org.ovirt.engine.core.bll.UploadStreamCommand] (DefaultQuartzScheduler_Worker-46) [233b68fb] Lock freed to object EngineLock [exclusiveLocks= key: 07c061c2-77f4-4c02-8402-e4dff773b216 value: DISK >key: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 value: STORAGE >key: 90108fe1-56a4-4f0e-b301-fa0ef46ba717 value: DISK >, sharedLocks= key: 00000002-0002-0002-0002-0000000001d9 value: OVF_UPDATE >] >2014-07-30 11:24:20,284 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (DefaultQuartzScheduler_Worker-46) [233b68fb] START, SetVolumeDescriptionVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, storageDomainId = 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752, imageGroupId = 90108fe1-56a4-4f0e-b301-fa0ef46ba717, imageId = 8596469e-37df-4d9e-9793-d238238131f5), log id: 428e56f1 >2014-07-30 11:24:20,286 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (DefaultQuartzScheduler_Worker-46) [233b68fb] -- executeIrsBrokerCommand: calling 'setVolumeDescription' >2014-07-30 11:24:20,289 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (DefaultQuartzScheduler_Worker-46) [233b68fb] -- setVolumeDescription parameters: > spUUID=00000002-0002-0002-0002-0000000001d9 > sdUUID=71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 > imageGroupGUID=90108fe1-56a4-4f0e-b301-fa0ef46ba717 > volUUID=8596469e-37df-4d9e-9793-d238238131f5 > description={"Updated":true,"Disk Description":"OVF_STORE","Storage Domains":[{"uuid":"71f82fcd-4fa6-4f56-b228-f4fa4bd4f752"}],"Last Updated":"Wed Jul 30 11:24:01 CEST 2014","Size":20480} > >2014-07-30 11:24:20,599 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (DefaultQuartzScheduler_Worker-46) [233b68fb] FINISH, SetVolumeDescriptionVDSCommand, log id: 428e56f1 >2014-07-30 11:24:20,607 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (DefaultQuartzScheduler_Worker-46) [233b68fb] START, SetVolumeDescriptionVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, storageDomainId = 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752, imageGroupId = 07c061c2-77f4-4c02-8402-e4dff773b216, imageId = 1f5f82d1-85c2-4386-8764-02bab6dcbc21), log id: 5804e57a >2014-07-30 11:24:20,609 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (DefaultQuartzScheduler_Worker-46) [233b68fb] -- executeIrsBrokerCommand: calling 'setVolumeDescription' >2014-07-30 11:24:20,610 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (DefaultQuartzScheduler_Worker-46) [233b68fb] -- setVolumeDescription parameters: > spUUID=00000002-0002-0002-0002-0000000001d9 > sdUUID=71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 > imageGroupGUID=07c061c2-77f4-4c02-8402-e4dff773b216 > volUUID=1f5f82d1-85c2-4386-8764-02bab6dcbc21 > description={"Updated":false,"Disk Description":"OVF_STORE","Storage Domains":[{"uuid":"71f82fcd-4fa6-4f56-b228-f4fa4bd4f752"}],"Last Updated":"Fri Jul 25 14:23:55 CEST 2014"} > >2014-07-30 11:24:20,963 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (DefaultQuartzScheduler_Worker-46) [233b68fb] FINISH, SetVolumeDescriptionVDSCommand, log id: 5804e57a >2014-07-30 11:24:20,988 INFO [org.ovirt.engine.core.bll.UploadStreamCommand] (DefaultQuartzScheduler_Worker-46) [56f1581c] Running command: UploadStreamCommand internal: true. Entities affected : ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage >2014-07-30 11:24:20,990 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.UploadStreamVDSCommand] (DefaultQuartzScheduler_Worker-46) [56f1581c] START, UploadStreamVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 7572bf6 >2014-07-30 11:24:20,991 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.UploadStreamVDSCommand] (DefaultQuartzScheduler_Worker-46) [56f1581c] -- executeVdsBrokerCommand: >2014-07-30 11:24:20,992 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.UploadStreamVDSCommand] (DefaultQuartzScheduler_Worker-46) [56f1581c] -- parameters: > spUUID=00000002-0002-0002-0002-0000000001d9 > sdUUID=71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 > imageGUID=07c061c2-77f4-4c02-8402-e4dff773b216 > volUUID=1f5f82d1-85c2-4386-8764-02bab6dcbc21 > size=20,480 > >2014-07-30 11:24:21,845 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.UploadStreamVDSCommand] (DefaultQuartzScheduler_Worker-46) [56f1581c] FINISH, UploadStreamVDSCommand, log id: 7572bf6 >2014-07-30 11:24:21,848 INFO [org.ovirt.engine.core.bll.CommandMultiAsyncTasks] (DefaultQuartzScheduler_Worker-46) [56f1581c] CommandMultiAsyncTasks::AttachTask: Attaching task 57ab43e6-b1ba-4a76-97af-26b0ed6e5559 to command 03335b91-a86c-46c6-8274-c21b3b5a9179. >2014-07-30 11:24:21,854 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (DefaultQuartzScheduler_Worker-46) [56f1581c] Adding task 57ab43e6-b1ba-4a76-97af-26b0ed6e5559 (Parent Command ProcessOvfUpdateForStorageDomain, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters), polling hasn't started yet.. >2014-07-30 11:24:21,856 WARN [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (DefaultQuartzScheduler_Worker-46) [56f1581c] Trying to release a shared lock for key: 00000002-0002-0002-0002-0000000001d9OVF_UPDATE , but lock does not exist >2014-07-30 11:24:21,857 WARN [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (DefaultQuartzScheduler_Worker-46) [56f1581c] Trying to release exclusive lock which does not exist, lock key: 07c061c2-77f4-4c02-8402-e4dff773b216DISK >2014-07-30 11:24:21,858 WARN [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (DefaultQuartzScheduler_Worker-46) [56f1581c] Trying to release exclusive lock which does not exist, lock key: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752STORAGE >2014-07-30 11:24:21,859 WARN [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (DefaultQuartzScheduler_Worker-46) [56f1581c] Trying to release exclusive lock which does not exist, lock key: 90108fe1-56a4-4f0e-b301-fa0ef46ba717DISK >2014-07-30 11:24:21,860 INFO [org.ovirt.engine.core.bll.UploadStreamCommand] (DefaultQuartzScheduler_Worker-46) [56f1581c] Lock freed to object EngineLock [exclusiveLocks= key: 07c061c2-77f4-4c02-8402-e4dff773b216 value: DISK >key: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 value: STORAGE >key: 90108fe1-56a4-4f0e-b301-fa0ef46ba717 value: DISK >, sharedLocks= key: 00000002-0002-0002-0002-0000000001d9 value: OVF_UPDATE >] >2014-07-30 11:24:21,864 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (DefaultQuartzScheduler_Worker-46) [56f1581c] START, SetVolumeDescriptionVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, storageDomainId = 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752, imageGroupId = 07c061c2-77f4-4c02-8402-e4dff773b216, imageId = 1f5f82d1-85c2-4386-8764-02bab6dcbc21), log id: 7976e0fd >2014-07-30 11:24:21,866 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (DefaultQuartzScheduler_Worker-46) [56f1581c] -- executeIrsBrokerCommand: calling 'setVolumeDescription' >2014-07-30 11:24:21,867 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (DefaultQuartzScheduler_Worker-46) [56f1581c] -- setVolumeDescription parameters: > spUUID=00000002-0002-0002-0002-0000000001d9 > sdUUID=71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 > imageGroupGUID=07c061c2-77f4-4c02-8402-e4dff773b216 > volUUID=1f5f82d1-85c2-4386-8764-02bab6dcbc21 > description={"Updated":true,"Disk Description":"OVF_STORE","Storage Domains":[{"uuid":"71f82fcd-4fa6-4f56-b228-f4fa4bd4f752"}],"Last Updated":"Wed Jul 30 11:24:01 CEST 2014","Size":20480} > >2014-07-30 11:24:21,990 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (DefaultQuartzScheduler_Worker-46) [56f1581c] FINISH, SetVolumeDescriptionVDSCommand, log id: 7976e0fd >2014-07-30 11:24:21,994 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (DefaultQuartzScheduler_Worker-46) [56f1581c] BaseAsyncTask::startPollingTask: Starting to poll task 13cca31c-d228-4247-a3cd-c324ae05433e. >2014-07-30 11:24:21,995 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (DefaultQuartzScheduler_Worker-46) [56f1581c] BaseAsyncTask::startPollingTask: Starting to poll task 57ab43e6-b1ba-4a76-97af-26b0ed6e5559. >2014-07-30 11:24:23,212 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (DefaultQuartzScheduler_Worker-49) Polling and updating Async Tasks: 2 tasks, 2 tasks to poll now >2014-07-30 11:24:23,218 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (DefaultQuartzScheduler_Worker-49) SPMAsyncTask::PollTask: Polling task 57ab43e6-b1ba-4a76-97af-26b0ed6e5559 (Parent Command ProcessOvfUpdateForStorageDomain, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) returned status finished, result 'success'. >2014-07-30 11:24:23,220 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (DefaultQuartzScheduler_Worker-49) BaseAsyncTask::OnTaskEndSuccess: Task 57ab43e6-b1ba-4a76-97af-26b0ed6e5559 (Parent Command ProcessOvfUpdateForStorageDomain, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) ended successfully. >2014-07-30 11:24:23,222 INFO [org.ovirt.engine.core.bll.CommandMultiAsyncTasks] (DefaultQuartzScheduler_Worker-49) Task with DB Task ID 0bb51d5e-c935-4fef-88aa-92f818eb4a10 and VDSM Task ID 13cca31c-d228-4247-a3cd-c324ae05433e is in state Polling. End action for command 03335b91-a86c-46c6-8274-c21b3b5a9179 will proceed when all the entitys tasks are completed. >2014-07-30 11:24:23,224 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (DefaultQuartzScheduler_Worker-49) SPMAsyncTask::PollTask: Polling task 13cca31c-d228-4247-a3cd-c324ae05433e (Parent Command ProcessOvfUpdateForStorageDomain, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) returned status finished, result 'success'. >2014-07-30 11:24:23,226 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (DefaultQuartzScheduler_Worker-49) BaseAsyncTask::OnTaskEndSuccess: Task 13cca31c-d228-4247-a3cd-c324ae05433e (Parent Command ProcessOvfUpdateForStorageDomain, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) ended successfully. >2014-07-30 11:24:23,227 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (DefaultQuartzScheduler_Worker-49) CommandAsyncTask::EndActionIfNecessary: All tasks of command 03335b91-a86c-46c6-8274-c21b3b5a9179 has ended -> executing endAction >2014-07-30 11:24:23,228 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (DefaultQuartzScheduler_Worker-49) CommandAsyncTask::endAction: Ending action for 2 tasks (command ID: 03335b91-a86c-46c6-8274-c21b3b5a9179): calling endAction . >2014-07-30 11:24:23,230 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-8-thread-21) CommandAsyncTask::EndCommandAction [within thread] context: Attempting to endAction ProcessOvfUpdateForStorageDomain, executionIndex: 0 >2014-07-30 11:24:23,239 INFO [org.ovirt.engine.core.bll.ProcessOvfUpdateForStorageDomainCommand] (org.ovirt.thread.pool-8-thread-21) [7a2d8c73] Ending command successfully: org.ovirt.engine.core.bll.ProcessOvfUpdateForStorageDomainCommand >2014-07-30 11:24:23,242 WARN [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (org.ovirt.thread.pool-8-thread-21) [7a2d8c73] Trying to release a shared lock for key: 00000002-0002-0002-0002-0000000001d9OVF_UPDATE , but lock does not exist >2014-07-30 11:24:23,244 WARN [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (org.ovirt.thread.pool-8-thread-21) [7a2d8c73] Trying to release exclusive lock which does not exist, lock key: 07c061c2-77f4-4c02-8402-e4dff773b216DISK >2014-07-30 11:24:23,246 WARN [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (org.ovirt.thread.pool-8-thread-21) [7a2d8c73] Trying to release exclusive lock which does not exist, lock key: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752STORAGE >2014-07-30 11:24:23,247 WARN [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (org.ovirt.thread.pool-8-thread-21) [7a2d8c73] Trying to release exclusive lock which does not exist, lock key: 90108fe1-56a4-4f0e-b301-fa0ef46ba717DISK >2014-07-30 11:24:23,248 INFO [org.ovirt.engine.core.bll.ProcessOvfUpdateForStorageDomainCommand] (org.ovirt.thread.pool-8-thread-21) [7a2d8c73] Lock freed to object EngineLock [exclusiveLocks= key: 07c061c2-77f4-4c02-8402-e4dff773b216 value: DISK >key: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 value: STORAGE >key: 90108fe1-56a4-4f0e-b301-fa0ef46ba717 value: DISK >, sharedLocks= key: 00000002-0002-0002-0002-0000000001d9 value: OVF_UPDATE >] >2014-07-30 11:24:23,250 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-8-thread-21) [7a2d8c73] CommandAsyncTask::HandleEndActionResult [within thread]: endAction for action type ProcessOvfUpdateForStorageDomain completed, handling the result. >2014-07-30 11:24:23,252 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-8-thread-21) [7a2d8c73] CommandAsyncTask::HandleEndActionResult [within thread]: endAction for action type ProcessOvfUpdateForStorageDomain succeeded, clearing tasks. >2014-07-30 11:24:23,253 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (org.ovirt.thread.pool-8-thread-21) [7a2d8c73] SPMAsyncTask::ClearAsyncTask: Attempting to clear task 57ab43e6-b1ba-4a76-97af-26b0ed6e5559 >2014-07-30 11:24:23,255 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-21) [7a2d8c73] START, SPMClearTaskVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, taskId = 57ab43e6-b1ba-4a76-97af-26b0ed6e5559), log id: 6ca7ec2 >2014-07-30 11:24:23,259 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-21) [7a2d8c73] START, HSMClearTaskVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, taskId=57ab43e6-b1ba-4a76-97af-26b0ed6e5559), log id: 5847f609 >2014-07-30 11:24:23,298 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-21) [7a2d8c73] FINISH, HSMClearTaskVDSCommand, log id: 5847f609 >2014-07-30 11:24:23,299 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-21) [7a2d8c73] FINISH, SPMClearTaskVDSCommand, log id: 6ca7ec2 >2014-07-30 11:24:23,305 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (org.ovirt.thread.pool-8-thread-21) [7a2d8c73] BaseAsyncTask::RemoveTaskFromDB: Removed task 57ab43e6-b1ba-4a76-97af-26b0ed6e5559 from DataBase >2014-07-30 11:24:23,306 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (org.ovirt.thread.pool-8-thread-21) [7a2d8c73] SPMAsyncTask::ClearAsyncTask: Attempting to clear task 13cca31c-d228-4247-a3cd-c324ae05433e >2014-07-30 11:24:23,307 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-21) [7a2d8c73] START, SPMClearTaskVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, taskId = 13cca31c-d228-4247-a3cd-c324ae05433e), log id: 3b47f7bf >2014-07-30 11:24:23,309 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-21) [7a2d8c73] START, HSMClearTaskVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, taskId=13cca31c-d228-4247-a3cd-c324ae05433e), log id: 71da3b00 >2014-07-30 11:24:23,379 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-21) [7a2d8c73] FINISH, HSMClearTaskVDSCommand, log id: 71da3b00 >2014-07-30 11:24:23,379 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-21) [7a2d8c73] FINISH, SPMClearTaskVDSCommand, log id: 3b47f7bf >2014-07-30 11:24:23,385 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (org.ovirt.thread.pool-8-thread-21) [7a2d8c73] BaseAsyncTask::RemoveTaskFromDB: Removed task 13cca31c-d228-4247-a3cd-c324ae05433e from DataBase >2014-07-30 11:24:23,386 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-8-thread-21) [7a2d8c73] CommandAsyncTask::HandleEndActionResult [within thread]: Removing CommandMultiAsyncTasks object for entity 03335b91-a86c-46c6-8274-c21b3b5a9179 >2014-07-30 11:25:29,087 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (DefaultQuartzScheduler_Worker-6) [4494a716] Setting new tasks map. The map contains now 0 tasks >2014-07-30 11:25:29,089 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (DefaultQuartzScheduler_Worker-6) [4494a716] Cleared all tasks of pool 00000002-0002-0002-0002-0000000001d9. >2014-07-30 12:23:55,123 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-78) Principal admin::internal synchronized >2014-07-30 12:24:21,996 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-44) [29bdffa0] Attempting to update VMs/Templates Ovf. >2014-07-30 12:28:54,585 WARN [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-8-thread-42) Executing a command: java.util.concurrent.FutureTask , but note that there are 1 tasks in the queue. >2014-07-30 12:28:54,589 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-22) [4a2f7e33] START, GetFileStatsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 523c8e76 >2014-07-30 12:28:54,595 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-42) [4a2f7e33] START, GetFileStatsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 21656ec0 >2014-07-30 12:28:54,895 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-22) [4a2f7e33] FINISH, GetFileStatsVDSCommand, return: {winxp.vfd={status=0, ctime=1403850000.0, size=1474560}, virtio-win-1.7.1_x86.vfd={status=0, ctime=1403850000.0, size=2949120}, virtio-win-1.7.1_amd64.vfd={status=0, ctime=1403850000.0, size=2949120}, win2003-32-unattended.vfd={status=0, ctime=1403850000.0, size=1474560}, winXP-Pro-SP2-32-unattended.vfd={status=0, ctime=1403850000.0, size=1474560}, win2000-Pro-unattended.vfd={status=0, ctime=1403850000.0, size=1474560}, win2k3x64.vfd={status=0, ctime=1403850000.0, size=1474560}, virtio-win_amd64.vfd={status=0, ctime=1403850000.0, size=2949120}, winXP-x64-unattended.vfd={status=0, ctime=1403850000.0, size=1474560}, virtio-win_x86.vfd={status=0, ctime=1403850000.0, size=2949120}, win2003-x64-unattended.vfd={status=0, ctime=1403850000.0, size=1474560}, floppy55.vfd={status=0, ctime=1403850000.0, size=1474560}, virtio-win-1.6.7_amd64.vfd={status=0, ctime=1403850000.0, size=2949120}}, log id: 523c8e76 >2014-07-30 12:28:54,912 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (org.ovirt.thread.pool-8-thread-22) [4a2f7e33] Finished automatic refresh process for Floppy file type with success, for storage domain id 0c78b4d6-ba00-4d3e-9f9f-65c7d5899d71. >2014-07-30 12:28:55,574 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-42) [4a2f7e33] FINISH, GetFileStatsVDSCommand, return: {deepin_2014_final_amd64.iso={status=0, ctime=1404720000.0, size=1581383680}, RHEV-toolsSetup_2.2_52998.iso={status=0, ctime=1403850000.0, size=105551872}, ubuntu-12.04.4-server-amd64.iso={status=0, ctime=1403850000.0, size=711983104}, RHEV-toolsSetup_3.2_15.iso={status=0, ctime=1403850000.0, size=264646656}, ubuntu-12.04-minicd.iso={status=0, ctime=1403850000.0, size=31457280}, rhev-hypervisor6-6.5-20140715.0.iso={status=0, ctime=1405590000.0, size=224395264}, RHEV-toolsSetup_3.1_8.iso={status=0, ctime=1403850000.0, size=220803072}, RHEV-toolsSetup_3.4_6.iso={status=0, ctime=1403850000.0, size=327026688}, virtio-win-1.6.7.iso={status=0, ctime=1403850000.0, size=97478656}, ovirt-guest-tools-3.5-2.iso={status=0, ctime=1403850000.0, size=76455936}, RHEV-toolsSetup_3.3_10.iso={status=0, ctime=1403850000.0, size=295870464}, RHEV-toolsSetup_3.0_34.iso={status=0, ctime=1403850000.0, size=133777408}, ubuntu-12.04-minimal-amd64.iso={status=0, ctime=1403850000.0, size=31457280}, debian-7.4.0-amd64-netinst.iso={status=0, ctime=1403850000.0, size=232783872}, RHEV-toolsSetup_3.4_7.iso={status=0, ctime=1403850000.0, size=327038976}, Windows2008-x64.iso={status=0, ctime=1403850000.0, size=2665758720}, en_windows_server_2012_r2_x64_dvd_2707946.iso={status=0, ctime=1403850000.0, size=4268605440}, elive_2.3.0_beta_hybrid.iso={status=0, ctime=1403850000.0, size=2187083776}, RHEV-toolsSetup_3.2_17.iso={status=0, ctime=1403850000.0, size=261437440}, RHEV-toolsSetup_3.3_7.iso={status=0, ctime=1403850000.0, size=295516160}, RHEV-toolsSetup_3.3_13.iso={status=0, ctime=1403850000.0, size=299448320}, en_windows_8_enterprise_x86_dvd_917587.iso={status=0, ctime=1403850000.0, size=2551154688}, RHEV-toolsSetup_3.2_16.iso={status=0, ctime=1403850000.0, size=261437440}, SLES-11-SP3-DVD-x86_64-GM-DVD2.iso={status=0, ctime=1403850000.0, size=5311318016}, centos.iso={status=0, ctime=1403850000.0, size=4467982336}, en-gb_windows_8.1_enterprise_with_update_x64_dvd_4048611.iso={status=0, ctime=1403850000.0, size=3964362752}, ubuntu-14.04-minimal-amd64.iso={status=0, ctime=1403850000.0, size=38797312}, RHEV-toolsSetup_3.4_1.iso={status=0, ctime=1403850000.0, size=326516736}, Fedora-Live-LXDE-x86_64-20-1.iso={status=0, ctime=1403850000.0, size=663748608}, ubuntu-13.10-server-amd64.iso={status=0, ctime=1403850000.0, size=704643072}, IT_WINXP_PRO_VL_ISO.iso={status=0, ctime=1403850000.0, size=515557376}, RHEV-toolsSetup_3.4_8.iso={status=0, ctime=1403850000.0, size=327120896}, ubuntu-14.04-server-amd64.iso={status=0, ctime=1403850000.0, size=591396864}, RHEV-toolsSetup_3.0_39.iso={status=0, ctime=1403850000.0, size=134195200}, ubuntu-13.10-minimal-amd64.iso={status=0, ctime=1403850000.0, size=37748736}, en_windows_7_enterprise_x86_dvd_x15-70745.iso={status=0, ctime=1403850000.0, size=2400401408}, RHEV-toolsSetup_3.0_37_ic156.iso={status=0, ctime=1403850000.0, size=133894144}, RHEV-toolsSetup_3.1_12.iso={status=0, ctime=1403850000.0, size=224448512}, EnglishWindowsServer2003R2withSP2VLPStandard64bitDISC1.iso={status=0, ctime=1403850000.0, size=652234752}, RHEV-toolsSetup_3.4_2.el6ev.iso={status=0, ctime=1403850000.0, size=326518784}, MUI_Windows_server_2003_r2_cd1.iso={status=0, ctime=1403850000.0, size=638373888}, openSUSE-13.1-NET-x86_64.iso={status=0, ctime=1403850000.0, size=297795584}, openSUSE-12.3-NET-x86_64.iso={status=0, ctime=1403850000.0, size=252706816}, install55.iso={status=0, ctime=1403850000.0, size=242722816}, RHEV-toolsSetup_3.4_4.el6ev.iso={status=0, ctime=1403850000.0, size=326424576}, FreeBSD-9.2-RELEASE-amd64-bootonly.iso={status=0, ctime=1404200000.0, size=172070912}, ubuntu-13.04-server-amd64.iso={status=0, ctime=1403850000.0, size=735051776}, en_windows_7_home_basic_with_sp1_x86_dvd_u_676470.iso={status=0, ctime=1403850000.0, size=2564476928}, es_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-74009.iso={status=0, ctime=1404390000.0, size=619194368}, RHEV-toolsSetup_3.3_6.iso={status=0, ctime=1403850000.0, size=295997440}, ovirt-guest-tools-3.5_5.iso={status=0, ctime=1406200000.0, size=106803200}, rhev-hypervisor6-6.5-20140112.0.iso={status=0, ctime=1403850000.0, size=226492416}, RHEV-toolsSetup_3.3_3.iso={status=0, ctime=1403850000.0, size=295911424}, RHEV-toolsSetup_3.4_5.el6ev.iso={status=0, ctime=1403850000.0, size=326780928}, fakeISO1.iso={status=0, ctime=1403850000.0, size=0}, es_windows_xp_professional_with_service_pack_3_x86_cd_x14-80488.iso={status=0, ctime=1404390000.0, size=619196416}, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso={status=0, ctime=1403850000.0, size=2996799488}, MX-14.2-pae.iso={status=0, ctime=1405060000.0, size=732954624}, RHEV-toolsSetup_2.3_395.iso={status=0, ctime=1403850000.0, size=133640192}, SLES-11-SP3-DVD-i586-GM-DVD1.iso={status=0, ctime=1403850000.0, size=3221225472}, ovirt-node-iso-3.1.0-0.999.999.20140317175137gitba60c851296.el6.iso={status=0, ctime=1406710000.0, size=211812352}, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso={status=0, ctime=1403850000.0, size=617754624}, RHEV-toolsSetup_2.3_359.iso={status=0, ctime=1403850000.0, size=133640192}, RHEV-toolsSetup_3.3_9.iso={status=0, ctime=1403850000.0, size=295872512}, FreeBSD-10.0-RELEASE-amd64-bootonly.iso={status=0, ctime=1404220000.0, size=220153856}, Pinguy_OS_14.04.1-Mini-x86-64.iso={status=0, ctime=1403850000.0, size=1583349760}, RHEV-toolsSetup_2.3_360-fake.iso={status=0, ctime=1403850000.0, size=0}, SLES-11-SP3-DVD-x86_64-GM-DVD1.iso={status=0, ctime=1403850000.0, size=3362783232}, RHEV-toolsSetup_3.0_37.iso={status=0, ctime=1403850000.0, size=133894144}, WANemv2.3.iso={status=0, ctime=1403850000.0, size=398247936}, RHEV-toolsSetup_3.3_14.iso={status=0, ctime=1403850000.0, size=299448320}, RHEV-toolsSetup_2.2_53011.iso={status=0, ctime=1403850000.0, size=105551872}, ovirt-guest-tools-3.4_2.iso={status=0, ctime=1403850000.0, size=76455936}, RHEV-toolsSetup_3.1_14.iso={status=0, ctime=1403850000.0, size=242872320}, ovirt-guest-tools-3.4-2.iso={status=0, ctime=1403850000.0, size=76455936}, es_windows_7_enterprise_with_sp1_x64_dvd_u_677652.iso={status=0, ctime=1404390000.0, size=3110127616}, EnglishWindowsServer2003R2withSP2VLPStandard64bitDISC2.iso={status=0, ctime=1403850000.0, size=175163392}, fakeISO01.iso={status=0, ctime=1403850000.0, size=0}, en_windows_7_enterprise_x64_dvd_x15-70749.iso={status=0, ctime=1403850000.0, size=3121215488}, Windows2008-x86.iso={status=0, ctime=1403850000.0, size=1881141248}, RHEL-7.0-20131222.0-Server-x86_64-dvd1.iso={status=0, ctime=1403850000.0, size=4580179968}, systemrescuecd-x86-4.2.0.iso={status=0, ctime=1404370000.0, size=399329280}, RHEV-toolsSetup_3.2_18.iso={status=0, ctime=1403850000.0, size=261439488}, CentOS-6.5-x86_64-minimal.iso={status=0, ctime=1403850000.0, size=417333248}, RHEV-toolsSetup_3.1_9.iso={status=0, ctime=1403850000.0, size=220809216}, RHEV-toolsSetup_3.3_8.iso={status=0, ctime=1403850000.0, size=295516160}, en_windows_8_enterprise_x64_dvd_917522.iso={status=0, ctime=1403850000.0, size=3490912256}, ubuntu-10.04-minimal-amd64.iso={status=0, ctime=1403850000.0, size=14299136}, elementaryos-stable-amd64.20130810.iso={status=0, ctime=1403850000.0, size=727711744}, qwerty.iso={status=0, ctime=1403850000.0, size=49530880}, RHEV-toolsSetup_3.3_11.iso={status=0, ctime=1403850000.0, size=295870464}, fakeISO02.iso={status=0, ctime=1403850000.0, size=0}, RHEL6.3-20120613.2-Server-x86_64-DVD1.iso={status=0, ctime=1403850000.0, size=3679453184}, Windows2003-r2-std-32-bit.iso={status=0, ctime=1403850000.0, size=600563712}, slax-English-US-7.0.8-x86_64.iso={status=0, ctime=1403850000.0, size=236482560}, en_windows_server_2012_x64_dvd_915478.iso={status=0, ctime=1403850000.0, size=3695179776}}, log id: 21656ec0 >2014-07-30 12:28:55,687 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (org.ovirt.thread.pool-8-thread-42) [4a2f7e33] Finished automatic refresh process for ISO file type with success, for storage domain id 0c78b4d6-ba00-4d3e-9f9f-65c7d5899d71. >2014-07-30 13:23:55,131 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-62) [2e742a17] Principal admin::internal synchronized >2014-07-30 13:24:22,023 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-38) [1b9ef0f] Attempting to update VMs/Templates Ovf. >2014-07-30 13:28:55,697 WARN [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-8-thread-47) Executing a command: java.util.concurrent.FutureTask , but note that there are 1 tasks in the queue. >2014-07-30 13:28:55,701 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-42) START, GetFileStatsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 148e385 >2014-07-30 13:28:55,705 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-47) START, GetFileStatsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 135c372a >2014-07-30 13:28:56,449 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-42) FINISH, GetFileStatsVDSCommand, return: {deepin_2014_final_amd64.iso={status=0, ctime=1404720000.0, size=1581383680}, RHEV-toolsSetup_2.2_52998.iso={status=0, ctime=1403850000.0, size=105551872}, ubuntu-12.04.4-server-amd64.iso={status=0, ctime=1403850000.0, size=711983104}, RHEV-toolsSetup_3.2_15.iso={status=0, ctime=1403850000.0, size=264646656}, ubuntu-12.04-minicd.iso={status=0, ctime=1403850000.0, size=31457280}, rhev-hypervisor6-6.5-20140715.0.iso={status=0, ctime=1405590000.0, size=224395264}, RHEV-toolsSetup_3.1_8.iso={status=0, ctime=1403850000.0, size=220803072}, RHEV-toolsSetup_3.4_6.iso={status=0, ctime=1403850000.0, size=327026688}, virtio-win-1.6.7.iso={status=0, ctime=1403850000.0, size=97478656}, ovirt-guest-tools-3.5-2.iso={status=0, ctime=1403850000.0, size=76455936}, RHEV-toolsSetup_3.3_10.iso={status=0, ctime=1403850000.0, size=295870464}, RHEV-toolsSetup_3.0_34.iso={status=0, ctime=1403850000.0, size=133777408}, ubuntu-12.04-minimal-amd64.iso={status=0, ctime=1403850000.0, size=31457280}, debian-7.4.0-amd64-netinst.iso={status=0, ctime=1403850000.0, size=232783872}, RHEV-toolsSetup_3.4_7.iso={status=0, ctime=1403850000.0, size=327038976}, Windows2008-x64.iso={status=0, ctime=1403850000.0, size=2665758720}, en_windows_server_2012_r2_x64_dvd_2707946.iso={status=0, ctime=1403850000.0, size=4268605440}, elive_2.3.0_beta_hybrid.iso={status=0, ctime=1403850000.0, size=2187083776}, RHEV-toolsSetup_3.2_17.iso={status=0, ctime=1403850000.0, size=261437440}, RHEV-toolsSetup_3.3_7.iso={status=0, ctime=1403850000.0, size=295516160}, RHEV-toolsSetup_3.3_13.iso={status=0, ctime=1403850000.0, size=299448320}, en_windows_8_enterprise_x86_dvd_917587.iso={status=0, ctime=1403850000.0, size=2551154688}, RHEV-toolsSetup_3.2_16.iso={status=0, ctime=1403850000.0, size=261437440}, SLES-11-SP3-DVD-x86_64-GM-DVD2.iso={status=0, ctime=1403850000.0, size=5311318016}, centos.iso={status=0, ctime=1403850000.0, size=4467982336}, en-gb_windows_8.1_enterprise_with_update_x64_dvd_4048611.iso={status=0, ctime=1403850000.0, size=3964362752}, ubuntu-14.04-minimal-amd64.iso={status=0, ctime=1403850000.0, size=38797312}, RHEV-toolsSetup_3.4_1.iso={status=0, ctime=1403850000.0, size=326516736}, Fedora-Live-LXDE-x86_64-20-1.iso={status=0, ctime=1403850000.0, size=663748608}, ubuntu-13.10-server-amd64.iso={status=0, ctime=1403850000.0, size=704643072}, IT_WINXP_PRO_VL_ISO.iso={status=0, ctime=1403850000.0, size=515557376}, RHEV-toolsSetup_3.4_8.iso={status=0, ctime=1403850000.0, size=327120896}, ubuntu-14.04-server-amd64.iso={status=0, ctime=1403850000.0, size=591396864}, RHEV-toolsSetup_3.0_39.iso={status=0, ctime=1403850000.0, size=134195200}, ubuntu-13.10-minimal-amd64.iso={status=0, ctime=1403850000.0, size=37748736}, en_windows_7_enterprise_x86_dvd_x15-70745.iso={status=0, ctime=1403850000.0, size=2400401408}, RHEV-toolsSetup_3.0_37_ic156.iso={status=0, ctime=1403850000.0, size=133894144}, RHEV-toolsSetup_3.1_12.iso={status=0, ctime=1403850000.0, size=224448512}, EnglishWindowsServer2003R2withSP2VLPStandard64bitDISC1.iso={status=0, ctime=1403850000.0, size=652234752}, RHEV-toolsSetup_3.4_2.el6ev.iso={status=0, ctime=1403850000.0, size=326518784}, MUI_Windows_server_2003_r2_cd1.iso={status=0, ctime=1403850000.0, size=638373888}, openSUSE-13.1-NET-x86_64.iso={status=0, ctime=1403850000.0, size=297795584}, openSUSE-12.3-NET-x86_64.iso={status=0, ctime=1403850000.0, size=252706816}, install55.iso={status=0, ctime=1403850000.0, size=242722816}, RHEV-toolsSetup_3.4_4.el6ev.iso={status=0, ctime=1403850000.0, size=326424576}, FreeBSD-9.2-RELEASE-amd64-bootonly.iso={status=0, ctime=1404200000.0, size=172070912}, ubuntu-13.04-server-amd64.iso={status=0, ctime=1403850000.0, size=735051776}, en_windows_7_home_basic_with_sp1_x86_dvd_u_676470.iso={status=0, ctime=1403850000.0, size=2564476928}, es_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-74009.iso={status=0, ctime=1404390000.0, size=619194368}, RHEV-toolsSetup_3.3_6.iso={status=0, ctime=1403850000.0, size=295997440}, ovirt-guest-tools-3.5_5.iso={status=0, ctime=1406200000.0, size=106803200}, rhev-hypervisor6-6.5-20140112.0.iso={status=0, ctime=1403850000.0, size=226492416}, RHEV-toolsSetup_3.3_3.iso={status=0, ctime=1403850000.0, size=295911424}, RHEV-toolsSetup_3.4_5.el6ev.iso={status=0, ctime=1403850000.0, size=326780928}, fakeISO1.iso={status=0, ctime=1403850000.0, size=0}, es_windows_xp_professional_with_service_pack_3_x86_cd_x14-80488.iso={status=0, ctime=1404390000.0, size=619196416}, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso={status=0, ctime=1403850000.0, size=2996799488}, MX-14.2-pae.iso={status=0, ctime=1405060000.0, size=732954624}, RHEV-toolsSetup_2.3_395.iso={status=0, ctime=1403850000.0, size=133640192}, SLES-11-SP3-DVD-i586-GM-DVD1.iso={status=0, ctime=1403850000.0, size=3221225472}, ovirt-node-iso-3.1.0-0.999.999.20140317175137gitba60c851296.el6.iso={status=0, ctime=1406710000.0, size=211812352}, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso={status=0, ctime=1403850000.0, size=617754624}, RHEV-toolsSetup_2.3_359.iso={status=0, ctime=1403850000.0, size=133640192}, RHEV-toolsSetup_3.3_9.iso={status=0, ctime=1403850000.0, size=295872512}, FreeBSD-10.0-RELEASE-amd64-bootonly.iso={status=0, ctime=1404220000.0, size=220153856}, Pinguy_OS_14.04.1-Mini-x86-64.iso={status=0, ctime=1403850000.0, size=1583349760}, RHEV-toolsSetup_2.3_360-fake.iso={status=0, ctime=1403850000.0, size=0}, SLES-11-SP3-DVD-x86_64-GM-DVD1.iso={status=0, ctime=1403850000.0, size=3362783232}, RHEV-toolsSetup_3.0_37.iso={status=0, ctime=1403850000.0, size=133894144}, WANemv2.3.iso={status=0, ctime=1403850000.0, size=398247936}, RHEV-toolsSetup_3.3_14.iso={status=0, ctime=1403850000.0, size=299448320}, RHEV-toolsSetup_2.2_53011.iso={status=0, ctime=1403850000.0, size=105551872}, ovirt-guest-tools-3.4_2.iso={status=0, ctime=1403850000.0, size=76455936}, RHEV-toolsSetup_3.1_14.iso={status=0, ctime=1403850000.0, size=242872320}, ovirt-guest-tools-3.4-2.iso={status=0, ctime=1403850000.0, size=76455936}, es_windows_7_enterprise_with_sp1_x64_dvd_u_677652.iso={status=0, ctime=1404390000.0, size=3110127616}, EnglishWindowsServer2003R2withSP2VLPStandard64bitDISC2.iso={status=0, ctime=1403850000.0, size=175163392}, fakeISO01.iso={status=0, ctime=1403850000.0, size=0}, en_windows_7_enterprise_x64_dvd_x15-70749.iso={status=0, ctime=1403850000.0, size=3121215488}, Windows2008-x86.iso={status=0, ctime=1403850000.0, size=1881141248}, RHEL-7.0-20131222.0-Server-x86_64-dvd1.iso={status=0, ctime=1403850000.0, size=4580179968}, systemrescuecd-x86-4.2.0.iso={status=0, ctime=1404370000.0, size=399329280}, RHEV-toolsSetup_3.2_18.iso={status=0, ctime=1403850000.0, size=261439488}, CentOS-6.5-x86_64-minimal.iso={status=0, ctime=1403850000.0, size=417333248}, RHEV-toolsSetup_3.1_9.iso={status=0, ctime=1403850000.0, size=220809216}, RHEV-toolsSetup_3.3_8.iso={status=0, ctime=1403850000.0, size=295516160}, en_windows_8_enterprise_x64_dvd_917522.iso={status=0, ctime=1403850000.0, size=3490912256}, ubuntu-10.04-minimal-amd64.iso={status=0, ctime=1403850000.0, size=14299136}, elementaryos-stable-amd64.20130810.iso={status=0, ctime=1403850000.0, size=727711744}, qwerty.iso={status=0, ctime=1403850000.0, size=49530880}, RHEV-toolsSetup_3.3_11.iso={status=0, ctime=1403850000.0, size=295870464}, fakeISO02.iso={status=0, ctime=1403850000.0, size=0}, RHEL6.3-20120613.2-Server-x86_64-DVD1.iso={status=0, ctime=1403850000.0, size=3679453184}, Windows2003-r2-std-32-bit.iso={status=0, ctime=1403850000.0, size=600563712}, slax-English-US-7.0.8-x86_64.iso={status=0, ctime=1403850000.0, size=236482560}, en_windows_server_2012_x64_dvd_915478.iso={status=0, ctime=1403850000.0, size=3695179776}}, log id: 148e385 >2014-07-30 13:28:56,549 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (org.ovirt.thread.pool-8-thread-42) Finished automatic refresh process for ISO file type with success, for storage domain id 0c78b4d6-ba00-4d3e-9f9f-65c7d5899d71. >2014-07-30 13:28:56,702 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-47) FINISH, GetFileStatsVDSCommand, return: {winxp.vfd={status=0, ctime=1403850000.0, size=1474560}, virtio-win-1.7.1_x86.vfd={status=0, ctime=1403850000.0, size=2949120}, virtio-win-1.7.1_amd64.vfd={status=0, ctime=1403850000.0, size=2949120}, win2003-32-unattended.vfd={status=0, ctime=1403850000.0, size=1474560}, winXP-Pro-SP2-32-unattended.vfd={status=0, ctime=1403850000.0, size=1474560}, win2000-Pro-unattended.vfd={status=0, ctime=1403850000.0, size=1474560}, win2k3x64.vfd={status=0, ctime=1403850000.0, size=1474560}, virtio-win_amd64.vfd={status=0, ctime=1403850000.0, size=2949120}, winXP-x64-unattended.vfd={status=0, ctime=1403850000.0, size=1474560}, virtio-win_x86.vfd={status=0, ctime=1403850000.0, size=2949120}, win2003-x64-unattended.vfd={status=0, ctime=1403850000.0, size=1474560}, floppy55.vfd={status=0, ctime=1403850000.0, size=1474560}, virtio-win-1.6.7_amd64.vfd={status=0, ctime=1403850000.0, size=2949120}}, log id: 135c372a >2014-07-30 13:28:56,714 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (org.ovirt.thread.pool-8-thread-47) Finished automatic refresh process for Floppy file type with success, for storage domain id 0c78b4d6-ba00-4d3e-9f9f-65c7d5899d71. >2014-07-30 13:44:54,175 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-40) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage domain 35 experienced a high latency of 5.54911 seconds from host rack. This may cause performance and functional issues. Please consult your Storage Administrator. >2014-07-30 13:57:42,073 WARN [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-8-thread-17) Executing a command: java.util.concurrent.FutureTask , but note that there are 1 tasks in the queue. >2014-07-30 14:23:55,137 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-16) Principal admin::internal synchronized >2014-07-30 14:24:22,059 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-60) Attempting to update VMs/Templates Ovf. >2014-07-30 14:25:13,526 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-34) domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 in problem. vds: rack >2014-07-30 14:25:13,529 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-34) domain 0c78b4d6-ba00-4d3e-9f9f-65c7d5899d71:str01-iso-shared in problem. vds: rack >2014-07-30 14:28:56,724 WARN [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-8-thread-27) Executing a command: java.util.concurrent.FutureTask , but note that there are 1 tasks in the queue. >2014-07-30 14:28:56,728 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-19) [7b19edd8] START, GetFileStatsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 21a51a33 >2014-07-30 14:28:56,731 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-27) [7b19edd8] START, GetFileStatsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 2e6a9841 >2014-07-30 14:30:13,531 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-35) starting processDomainRecovery for domain 0c78b4d6-ba00-4d3e-9f9f-65c7d5899d71:str01-iso-shared >2014-07-30 14:30:13,598 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-35) Domain 0c78b4d6-ba00-4d3e-9f9f-65c7d5899d71:str01-iso-shared was reported by all hosts in status UP as problematic. Moving the domain to NonOperational. >2014-07-30 14:30:13,627 INFO [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-35) [6c3a9186] Lock Acquired to object EngineLock [exclusiveLocks= key: 0c78b4d6-ba00-4d3e-9f9f-65c7d5899d71 value: STORAGE >, sharedLocks= ] >2014-07-30 14:30:13,637 INFO [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-35) [6c3a9186] Running command: DeactivateStorageDomainCommand internal: true. Entities affected : ID: 0c78b4d6-ba00-4d3e-9f9f-65c7d5899d71 Type: Storage >2014-07-30 14:30:13,651 INFO [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-35) [6c3a9186] Lock freed to object EngineLock [exclusiveLocks= key: 0c78b4d6-ba00-4d3e-9f9f-65c7d5899d71 value: STORAGE >, sharedLocks= ] >2014-07-30 14:30:13,665 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-35) [6c3a9186] Correlation ID: 6c3a9186, Job ID: abc6e47b-f0ed-4874-9b4b-8b6d316d0d5f, Call Stack: null, Custom Event ID: -1, Message: Storage Domain str01-iso-shared (Data Center Default) was deactivated by system because it's not visible by any of the hosts. >2014-07-30 14:30:13,673 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-35) [6c3a9186] Removing vds [] from the domain in maintenance cache >2014-07-30 14:30:13,674 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-35) [6c3a9186] Removing host(s) [] from hosts unseen domain report cache >2014-07-30 14:30:13,676 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-40) starting processDomainRecovery for domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 >2014-07-30 14:30:13,681 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-40) Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 was reported by all hosts in status UP as problematic. Not moving the domain to NonOperational because it is being reconstructed now. >2014-07-30 14:30:13,704 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-40) [2a1cf137] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage >2014-07-30 14:30:13,728 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-40) [2a1cf137] START, ResetIrsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, ignoreStopFailed = true), log id: a93288e >2014-07-30 14:30:13,732 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-40) [2a1cf137] START, SpmStopVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9), log id: 11255cc8 >2014-07-30 14:30:13,738 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-40) [2a1cf137] SpmStopVDSCommand::Stopping SPM on vds rack, pool id 00000002-0002-0002-0002-0000000001d9 >2014-07-30 14:30:23,458 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-7) Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2014-07-30 14:30:23,463 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-7) [2b0f39f8] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-07-30 14:30:23,480 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-7) [2b0f39f8] Correlation ID: 2b0f39f8, Call Stack: null, Custom Event ID: -1, Message: Invalid status on Data Center Default. Setting Data Center status to Non Responsive (On host rack, Error: Cannot access master storage domain). >2014-07-30 14:30:23,527 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-7) [2b0f39f8] Command org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand return value > >TaskStatusListReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=654, mMessage=Not SPM]] > >2014-07-30 14:30:23,529 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-7) [2b0f39f8] HostName = rack >2014-07-30 14:30:23,532 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-7) [2b0f39f8] Command HSMGetAllTasksStatusesVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b) execution failed. Exception: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM >2014-07-30 14:32:23,653 WARN [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-40) [2a1cf137] could not stop spm of pool 00000002-0002-0002-0002-0000000001d9 on vds 17f7fe56-5f32-40ef-b098-611b84484e0b - reason: org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcRunTimeException: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException >2014-07-30 14:32:23,655 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-7) [2b0f39f8] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: VDSNetworkException: org.apache.commons.httpclient.NoHttpResponseException: The server 10.34.62.208 failed to respond >2014-07-30 14:32:23,656 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-40) [2a1cf137] FINISH, SpmStopVDSCommand, log id: 11255cc8 >2014-07-30 14:32:23,658 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-7) [2b0f39f8] hostFromVds::selectedVds - rack, spmStatus returned null! >2014-07-30 14:32:23,665 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-40) [2a1cf137] FINISH, ResetIrsVDSCommand, log id: a93288e >2014-07-30 14:32:23,670 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (org.ovirt.thread.pool-8-thread-27) [7b19edd8] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-07-30 14:32:23,671 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-27) [7b19edd8] hostFromVds::selectedVds - rack, spmStatus returned null! >2014-07-30 14:32:23,672 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-27) [7b19edd8] FINISH, GetFileStatsVDSCommand, log id: 2e6a9841 >2014-07-30 14:32:23,675 WARN [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (org.ovirt.thread.pool-8-thread-27) [7b19edd8] The refresh process from VDSM, for Iso files failed. >2014-07-30 14:32:23,677 ERROR [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (org.ovirt.thread.pool-8-thread-27) [7b19edd8] org.ovirt.engine.core.common.errors.VdcBLLException: VdcBLLException: Cannot allocate IRS server (Failed with error IRS_REPOSITORY_NOT_FOUND and code 5009) >2014-07-30 14:32:23,680 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (org.ovirt.thread.pool-8-thread-19) [7b19edd8] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-07-30 14:32:23,683 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-19) [7b19edd8] hostFromVds::selectedVds - rack, spmStatus returned null! >2014-07-30 14:32:23,684 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-19) [7b19edd8] FINISH, GetFileStatsVDSCommand, log id: 21a51a33 >2014-07-30 14:32:23,685 WARN [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (org.ovirt.thread.pool-8-thread-19) [7b19edd8] The refresh process from VDSM, for Floppy files failed. >2014-07-30 14:32:23,685 ERROR [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (org.ovirt.thread.pool-8-thread-19) [7b19edd8] org.ovirt.engine.core.common.errors.VdcBLLException: VdcBLLException: Cannot allocate IRS server (Failed with error IRS_REPOSITORY_NOT_FOUND and code 5009) >2014-07-30 14:32:23,681 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (org.ovirt.thread.pool-8-thread-27) [7b19edd8] Finished automatic refresh process for ISO file type with failure, for storage domain id 0c78b4d6-ba00-4d3e-9f9f-65c7d5899d71. >2014-07-30 14:32:23,688 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (org.ovirt.thread.pool-8-thread-19) [7b19edd8] Finished automatic refresh process for Floppy file type with failure, for storage domain id 0c78b4d6-ba00-4d3e-9f9f-65c7d5899d71. >2014-07-30 14:32:23,690 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-40) [2a1cf137] Correlation ID: 2a1cf137, Job ID: 95754423-1a75-479b-9b6b-917dc1515c25, Call Stack: null, Custom Event ID: -1, Message: Failed to Reconstruct Master Domain for Data Center Default. >2014-07-30 14:32:23,701 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-40) [2a1cf137] Removing vds [] from the domain in maintenance cache >2014-07-30 14:32:23,703 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-40) [2a1cf137] Removing host(s) [] from hosts unseen domain report cache >2014-07-30 14:32:23,704 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-34) [58bfe349] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue >2014-07-30 14:32:23,706 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-34) [58bfe349] The following operation Event [eventType=DOMAINMONITORING, storagePoolId=00000002-0002-0002-0002-0000000001d9, domainId=null, vdsId=17f7fe56-5f32-40ef-b098-611b84484e0b, description=] was cancelled, because of reconstruct was run before >2014-07-30 14:32:23,738 ERROR [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (DefaultQuartzScheduler_Worker-58) [7b19edd8] The following storage domains had a problem retrieving data from VDSM ( Storage Pool Id: 00000002-0002-0002-0002-0000000001d9 Storage domain Id: 0c78b4d6-ba00-4d3e-9f9f-65c7d5899d71 File type: ISO) ( Storage Pool Id: 00000002-0002-0002-0002-0000000001d9 Storage domain Id: 0c78b4d6-ba00-4d3e-9f9f-65c7d5899d71 File type: Floppy) >2014-07-30 14:32:23,743 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-58) [7b19edd8] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Refresh image list failed for domain(s): str01-iso-shared (ISO file type) str01-iso-shared (Floppy file type). Please check domain activity. >2014-07-30 14:32:26,806 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ListVDSCommand] (DefaultQuartzScheduler_Worker-12) Command ListVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]) execution failed. Exception: VDSRecoveringException: Recovering from crash or Initializing >2014-07-30 14:32:26,808 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-12) Failed vds listing, vds = 17f7fe56-5f32-40ef-b098-611b84484e0b : rack, error = org.ovirt.engine.core.vdsbroker.vdsbroker.VDSRecoveringException: Recovering from crash or Initializing >2014-07-30 14:32:26,820 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-12) Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSRecoveringException: Recovering from crash or Initializing > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.vdsbroker.ListVDSCommand.executeVdsBrokerCommand(ListVDSCommand.java:24) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:406) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.fetchRunningVms(VdsUpdateRunTimeInfo.java:971) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVmStats(VdsUpdateRunTimeInfo.java:920) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:638) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:235) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Host rack is initializing. Message: Recovering from crash or Initializing >2014-07-30 14:32:26,827 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-12) Failed to refresh VDS , vds = 17f7fe56-5f32-40ef-b098-611b84484e0b : rack, error = Recovering from crash or Initializing, continuing. >2014-07-30 14:32:30,062 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-95) [72971053] START, GetHardwareInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]), log id: 60cf4a11 >2014-07-30 14:32:30,113 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-95) [72971053] FINISH, GetHardwareInfoVDSCommand, log id: 60cf4a11 >2014-07-30 14:32:30,146 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-95) [767bfeb3] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS >2014-07-30 14:32:30,155 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-95) [767bfeb3] Cancelling the recovery from crash timer for VDS 17f7fe56-5f32-40ef-b098-611b84484e0b because vds started initializing >2014-07-30 14:32:30,163 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-95) [374f1b92] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-07-30 14:32:30,171 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-95) [45b82729] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-07-30 14:32:30,186 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-95) [45b82729] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, storageType = NFS, connectionList = [{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 51e002a3-3547-4fbf-b3c9-6a4536cca8a8, connection: 10.34.63.204:/home/iso/shared, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 2c036780 >2014-07-30 14:32:30,709 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-95) [45b82729] FINISH, ConnectStorageServerVDSCommand, return: {51e002a3-3547-4fbf-b3c9-6a4536cca8a8=0, 5083f370-af09-457b-9b73-3440e8b65a07=0}, log id: 2c036780 >2014-07-30 14:32:30,711 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-95) [45b82729] Host rack storage connection was succeeded >2014-07-30 14:32:30,719 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-42) START, ConnectStoragePoolVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, masterVersion = 1), log id: 68fe8d22 >2014-07-30 14:32:33,666 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-40) Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 0c78b4d6-ba00-4d3e-9f9f-65c7d5899d71 status from Inactive to Unknown, reason : no reporting hosts >2014-07-30 14:32:33,670 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-40) Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Inactive to Unknown, reason : no reporting hosts >2014-07-30 14:33:37,028 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-42) FINISH, ConnectStoragePoolVDSCommand, log id: 68fe8d22 >2014-07-30 14:33:37,042 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-95) [45b82729] START, SetMOMPolicyParametersVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 9e0610 >2014-07-30 14:33:37,136 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-95) [45b82729] FINISH, SetMOMPolicyParametersVDSCommand, log id: 9e0610 >2014-07-30 14:33:37,143 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-95) [45b82729] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host rack was set to Up. >2014-07-30 14:33:37,149 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-95) [45b82729] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host rack. >2014-07-30 14:33:37,167 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-95) [ceef331] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS >2014-07-30 14:33:43,833 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-52) hostFromVds::selectedVds - rack, spmStatus Free, storage pool Default >2014-07-30 14:33:43,837 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-52) starting spm on vds rack, storage pool Default, prevId -1, LVER -1 >2014-07-30 14:33:43,840 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-52) START, SpmStartVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, prevId=-1, prevLVER=-1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 52394d74 >2014-07-30 14:33:43,888 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-52) spmStart polling started: taskId = 20f424e1-7097-4019-952c-2de62e8f5aee >2014-07-30 14:33:44,894 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-52) spmStart polling ended: taskId = 20f424e1-7097-4019-952c-2de62e8f5aee task status = finished >2014-07-30 14:33:44,950 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-52) spmStart polling ended, spm status: SPM >2014-07-30 14:33:44,953 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-52) START, HSMClearTaskVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, taskId=20f424e1-7097-4019-952c-2de62e8f5aee), log id: 731228b1 >2014-07-30 14:33:44,997 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-52) FINISH, HSMClearTaskVDSCommand, log id: 731228b1 >2014-07-30 14:33:44,998 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-52) FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@4ecef617, log id: 52394d74 >2014-07-30 14:33:45,008 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-52) Initialize Irs proxy from vds: 10.34.62.208 >2014-07-30 14:33:45,014 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-52) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208). >2014-07-30 14:33:45,016 WARN [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-8-thread-50) Executing a command: java.util.concurrent.FutureTask , but note that there are 1 tasks in the queue. >2014-07-30 14:33:45,030 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-50) START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 39420802 >2014-07-30 14:33:46,991 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-50) -- executeIrsBrokerCommand: Attempting on storage pool 00000002-0002-0002-0002-0000000001d9 >2014-07-30 14:33:46,993 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-50) START, HSMGetAllTasksInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 15a92800 >2014-07-30 14:33:46,998 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-50) FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 15a92800 >2014-07-30 14:33:46,999 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-50) FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 39420802 >2014-07-30 14:33:47,000 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (org.ovirt.thread.pool-8-thread-50) Discovered no tasks on Storage Pool Default >2014-07-30 15:08:18,836 INFO [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (ajp--127.0.0.1-8702-5) [20a126cf] Lock Acquired to object EngineLock [exclusiveLocks= key: 0c78b4d6-ba00-4d3e-9f9f-65c7d5899d71 value: STORAGE >, sharedLocks= ] >2014-07-30 15:08:18,932 INFO [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-27) [20a126cf] Running command: DeactivateStorageDomainCommand internal: false. Entities affected : ID: 0c78b4d6-ba00-4d3e-9f9f-65c7d5899d71 Type: Storage >2014-07-30 15:08:18,948 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.DeactivateStorageDomainVDSCommand] (org.ovirt.thread.pool-8-thread-27) [20a126cf] START, DeactivateStorageDomainVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, storageDomainId = 0c78b4d6-ba00-4d3e-9f9f-65c7d5899d71, masterDomainId = 00000000-0000-0000-0000-000000000000, masterVersion = 1), log id: 5b362203 >2014-07-30 15:08:18,960 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.DeactivateStorageDomainVDSCommand] (org.ovirt.thread.pool-8-thread-27) [20a126cf] FINISH, DeactivateStorageDomainVDSCommand, log id: 5b362203 >2014-07-30 15:08:18,962 INFO [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-27) [20a126cf] Lock freed to object EngineLock [exclusiveLocks= key: 0c78b4d6-ba00-4d3e-9f9f-65c7d5899d71 value: STORAGE >, sharedLocks= ] >2014-07-30 15:08:18,971 INFO [org.ovirt.engine.core.bll.storage.AfterDeactivateSingleAsyncOperation] (org.ovirt.thread.pool-8-thread-32) After deactivate treatment vds: rack,pool Default >2014-07-30 15:08:18,974 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-32) START, ConnectStoragePoolVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, masterVersion = 1), log id: 4bfd7af2 >2014-07-30 15:09:25,129 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-32) FINISH, ConnectStoragePoolVDSCommand, log id: 4bfd7af2 >2014-07-30 15:09:25,133 INFO [org.ovirt.engine.core.bll.storage.DisconnectStorageServerConnectionCommand] (org.ovirt.thread.pool-8-thread-27) [73b55b9e] Running command: DisconnectStorageServerConnectionCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2014-07-30 15:09:25,135 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-15) Adding domain 0c78b4d6-ba00-4d3e-9f9f-65c7d5899d71 to the domains in maintenance cache >2014-07-30 15:09:25,136 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-27) [73b55b9e] START, DisconnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: 51e002a3-3547-4fbf-b3c9-6a4536cca8a8, connection: 10.34.63.204:/home/iso/shared, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 43d446e2 >2014-07-30 15:09:25,190 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-50) Moving domain 0c78b4d6-ba00-4d3e-9f9f-65c7d5899d71 to maintenance >2014-07-30 15:09:25,197 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-45) Moving domain 0c78b4d6-ba00-4d3e-9f9f-65c7d5899d71 to maintenance >2014-07-30 15:09:25,203 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-35) Moving domain 0c78b4d6-ba00-4d3e-9f9f-65c7d5899d71 to maintenance >2014-07-30 15:09:25,208 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-30) Moving domain 0c78b4d6-ba00-4d3e-9f9f-65c7d5899d71 to maintenance >2014-07-30 15:09:25,214 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-48) Moving domain 0c78b4d6-ba00-4d3e-9f9f-65c7d5899d71 to maintenance >2014-07-30 15:09:25,362 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-27) [73b55b9e] FINISH, DisconnectStorageServerVDSCommand, return: {51e002a3-3547-4fbf-b3c9-6a4536cca8a8=0}, log id: 43d446e2 >2014-07-30 15:09:25,364 INFO [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-27) [73b55b9e] Domain 0c78b4d6-ba00-4d3e-9f9f-65c7d5899d71 will remain in PreparingForMaintenance status until deactivated on all hosts >2014-07-30 15:09:25,376 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-27) [73b55b9e] Correlation ID: 20a126cf, Job ID: 00122294-d65c-427d-bd65-7ec499615540, Call Stack: null, Custom Event ID: -1, Message: Storage Domain str01-iso-shared (Data Center Default) was deactivated by admin >2014-07-30 15:09:35,207 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-19) Moving domain 0c78b4d6-ba00-4d3e-9f9f-65c7d5899d71 to maintenance >2014-07-30 15:18:59,554 INFO [org.ovirt.engine.core.bll.storage.DetachStorageDomainFromPoolCommand] (org.ovirt.thread.pool-8-thread-44) [10039d15] Running command: DetachStorageDomainFromPoolCommand internal: false. Entities affected : ID: 0c78b4d6-ba00-4d3e-9f9f-65c7d5899d71 Type: Storage >2014-07-30 15:18:59,558 INFO [org.ovirt.engine.core.bll.storage.DetachStorageDomainFromPoolCommand] (org.ovirt.thread.pool-8-thread-44) [10039d15] Start detach storage domain >2014-07-30 15:18:59,563 INFO [org.ovirt.engine.core.bll.storage.DetachStorageDomainFromPoolCommand] (org.ovirt.thread.pool-8-thread-44) [10039d15] Detach storage domain: before connect >2014-07-30 15:18:59,567 INFO [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-8-thread-22) [1ed964c1] Running command: ConnectStorageToVdsCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2014-07-30 15:18:59,571 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-22) [1ed964c1] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: 51e002a3-3547-4fbf-b3c9-6a4536cca8a8, connection: 10.34.63.204:/home/iso/shared, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 2ecb99fb >2014-07-30 15:18:59,645 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-22) [1ed964c1] FINISH, ConnectStorageServerVDSCommand, return: {51e002a3-3547-4fbf-b3c9-6a4536cca8a8=0}, log id: 2ecb99fb >2014-07-30 15:18:59,647 INFO [org.ovirt.engine.core.bll.storage.DetachStorageDomainFromPoolCommand] (org.ovirt.thread.pool-8-thread-44) [10039d15] Detach storage domain: after connect >2014-07-30 15:18:59,649 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.DetachStorageDomainVDSCommand] (org.ovirt.thread.pool-8-thread-44) [10039d15] START, DetachStorageDomainVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, storageDomainId = 0c78b4d6-ba00-4d3e-9f9f-65c7d5899d71, masterDomainId = 00000000-0000-0000-0000-000000000000, masterVersion = 1, force = false), log id: 6eaeea5f >2014-07-30 15:18:59,665 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-47) Removing domain 0c78b4d6-ba00-4d3e-9f9f-65c7d5899d71 from the domains in maintenance cache >2014-07-30 15:19:02,945 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.DetachStorageDomainVDSCommand] (org.ovirt.thread.pool-8-thread-44) [10039d15] FINISH, DetachStorageDomainVDSCommand, log id: 6eaeea5f >2014-07-30 15:19:02,946 INFO [org.ovirt.engine.core.bll.storage.DetachStorageDomainFromPoolCommand] (org.ovirt.thread.pool-8-thread-44) [10039d15] Detach storage domain: after detach in vds >2014-07-30 15:19:02,954 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-18) START, ConnectStoragePoolVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, masterVersion = 1), log id: 51703187 >2014-07-30 15:19:18,055 INFO [org.ovirt.engine.core.bll.storage.RemoveStorageDomainCommand] (ajp--127.0.0.1-8702-5) [39bbddf7] Lock Acquired to object EngineLock [exclusiveLocks= key: 0c78b4d6-ba00-4d3e-9f9f-65c7d5899d71 value: STORAGE >, sharedLocks= ] >2014-07-30 15:19:18,063 INFO [org.ovirt.engine.core.bll.storage.RemoveStorageDomainCommand] (ajp--127.0.0.1-8702-5) [39bbddf7] Running command: RemoveStorageDomainCommand internal: false. Entities affected : ID: 0c78b4d6-ba00-4d3e-9f9f-65c7d5899d71 Type: Storage >2014-07-30 15:19:18,160 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-5) [39bbddf7] Correlation ID: 39bbddf7, Job ID: aa86eacb-39d8-4506-87a8-366ba61bc633, Call Stack: null, Custom Event ID: -1, Message: Storage Domain str01-iso-shared was removed by admin >2014-07-30 15:19:18,167 INFO [org.ovirt.engine.core.bll.storage.RemoveStorageDomainCommand] (ajp--127.0.0.1-8702-5) [39bbddf7] Lock freed to object EngineLock [exclusiveLocks= key: 0c78b4d6-ba00-4d3e-9f9f-65c7d5899d71 value: STORAGE >, sharedLocks= ] >2014-07-30 15:20:05,067 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-18) FINISH, ConnectStoragePoolVDSCommand, log id: 51703187 >2014-07-30 15:20:05,069 WARN [org.ovirt.engine.core.bll.storage.NFSStorageHelper] (org.ovirt.thread.pool-8-thread-18) Did not connect host: 17f7fe56-5f32-40ef-b098-611b84484e0b to storage domain: str01-iso-shared because connection for connectionId:51e002a3-3547-4fbf-b3c9-6a4536cca8a8 is null. >2014-07-30 15:20:05,071 INFO [org.ovirt.engine.core.bll.storage.DetachStorageDomainFromPoolCommand] (org.ovirt.thread.pool-8-thread-44) [10039d15] Detach storage domain: after disconnect storage >2014-07-30 15:20:05,167 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ResetISOPathVDSCommand] (org.ovirt.thread.pool-8-thread-44) [10039d15] START, ResetISOPathVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 39d1c3a1 >2014-07-30 15:20:05,168 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ResetISOPathVDSCommand] (org.ovirt.thread.pool-8-thread-44) [10039d15] FINISH, ResetISOPathVDSCommand, log id: 39d1c3a1 >2014-07-30 15:20:05,170 INFO [org.ovirt.engine.core.bll.storage.DetachStorageDomainFromPoolCommand] (org.ovirt.thread.pool-8-thread-44) [10039d15] End detach storage domain >2014-07-30 15:20:05,178 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-44) [10039d15] Correlation ID: 10039d15, Job ID: 6d926e1a-877e-4402-9f30-a71ac1f9db6c, Call Stack: null, Custom Event ID: -1, Message: Storage Domain str01-iso-shared was detached from Data Center Default by admin >2014-07-30 15:23:13,482 INFO [org.ovirt.engine.core.bll.storage.AttachStorageDomainToPoolCommand] (org.ovirt.thread.pool-8-thread-12) [63580ade] Running command: AttachStorageDomainToPoolCommand internal: false. Entities affected : ID: 63fe3516-4170-402a-9dc8-6d3eab1c7933 Type: Storage >2014-07-30 15:23:13,492 INFO [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-8-thread-50) [1286263f] Running command: ConnectStorageToVdsCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2014-07-30 15:23:13,496 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-50) [1286263f] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: 7fe036a2-2dfc-4565-b936-69a678cf0613, connection: pb-rh35.rhev.lab.eng.brq.redhat.com:/var/lib/exports/iso, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 6563eb38 >2014-07-30 15:23:55,145 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-55) [296dca52] Principal admin::internal synchronized >2014-07-30 15:24:22,086 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-11) Attempting to update VMs/Templates Ovf. >2014-07-30 15:25:23,566 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-50) [1286263f] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host rack to the Storage Domains ISO_DOMAIN. >2014-07-30 15:25:23,567 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-50) [1286263f] FINISH, ConnectStorageServerVDSCommand, return: {7fe036a2-2dfc-4565-b936-69a678cf0613=477}, log id: 6563eb38 >2014-07-30 15:25:23,570 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-50) [1286263f] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: The error message for connection pb-rh35.rhev.lab.eng.brq.redhat.com:/var/lib/exports/iso returned by VDSM was: Problem while trying to mount target >2014-07-30 15:25:23,572 ERROR [org.ovirt.engine.core.bll.storage.NFSStorageHelper] (org.ovirt.thread.pool-8-thread-50) [1286263f] The connection with details pb-rh35.rhev.lab.eng.brq.redhat.com:/var/lib/exports/iso failed because of error code 477 and error message is: problem while trying to mount target >2014-07-30 15:25:23,578 ERROR [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-8-thread-50) [1286263f] Transaction rolled-back for command: org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand. >2014-07-30 15:25:23,579 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.AttachStorageDomainVDSCommand] (org.ovirt.thread.pool-8-thread-12) [63580ade] START, AttachStorageDomainVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, storageDomainId = 63fe3516-4170-402a-9dc8-6d3eab1c7933), log id: b91988e >2014-07-30 15:27:24,136 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.AttachStorageDomainVDSCommand] (org.ovirt.thread.pool-8-thread-12) [63580ade] Failed in AttachStorageDomainVDS method >2014-07-30 15:27:24,146 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (org.ovirt.thread.pool-8-thread-12) [63580ade] IrsBroker::Failed::AttachStorageDomainVDS due to: IRSErrorException: IRSGenericException: IRSErrorException: Failed to AttachStorageDomainVDS, error = Storage domain does not exist: ('63fe3516-4170-402a-9dc8-6d3eab1c7933',), code = 358 >2014-07-30 15:27:24,156 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.AttachStorageDomainVDSCommand] (org.ovirt.thread.pool-8-thread-12) [63580ade] FINISH, AttachStorageDomainVDSCommand, log id: b91988e >2014-07-30 15:27:24,158 ERROR [org.ovirt.engine.core.bll.storage.AttachStorageDomainToPoolCommand] (org.ovirt.thread.pool-8-thread-12) [63580ade] Command org.ovirt.engine.core.bll.storage.AttachStorageDomainToPoolCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.irsbroker.IRSErrorException: IRSGenericException: IRSErrorException: Failed to AttachStorageDomainVDS, error = Storage domain does not exist: ('63fe3516-4170-402a-9dc8-6d3eab1c7933',), code = 358 (Failed with error StorageDomainDoesNotExist and code 358) >2014-07-30 15:27:24,163 INFO [org.ovirt.engine.core.bll.storage.AttachStorageDomainToPoolCommand] (org.ovirt.thread.pool-8-thread-12) [63580ade] Command [id=9295ce3e-dde8-4d91-ac4e-368395f4d41d]: Compensating NEW_ENTITY_ID of org.ovirt.engine.core.common.businessentities.StoragePoolIsoMap; snapshot: storagePoolId = 00000002-0002-0002-0002-0000000001d9, storageId = 63fe3516-4170-402a-9dc8-6d3eab1c7933. >2014-07-30 15:27:24,175 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-12) [63580ade] Correlation ID: 63580ade, Job ID: eed628df-27b6-4c89-9482-c76ff2b562c0, Call Stack: null, Custom Event ID: -1, Message: Failed to attach Storage Domain ISO_DOMAIN to Data Center Default. (User: admin) >2014-07-30 15:33:59,448 INFO [org.ovirt.engine.core.bll.storage.AttachStorageDomainToPoolCommand] (org.ovirt.thread.pool-8-thread-29) [4b747578] Running command: AttachStorageDomainToPoolCommand internal: false. Entities affected : ID: 63fe3516-4170-402a-9dc8-6d3eab1c7933 Type: Storage >2014-07-30 15:33:59,460 INFO [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-8-thread-20) [26e7de54] Running command: ConnectStorageToVdsCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2014-07-30 15:33:59,464 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-20) [26e7de54] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: 7fe036a2-2dfc-4565-b936-69a678cf0613, connection: pb-rh35.rhev.lab.eng.brq.redhat.com:/var/lib/exports/iso, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 747b3d4e >2014-07-30 15:36:09,528 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-20) [26e7de54] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host rack to the Storage Domains ISO_DOMAIN. >2014-07-30 15:36:09,529 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-20) [26e7de54] FINISH, ConnectStorageServerVDSCommand, return: {7fe036a2-2dfc-4565-b936-69a678cf0613=477}, log id: 747b3d4e >2014-07-30 15:36:09,532 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-20) [26e7de54] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: The error message for connection pb-rh35.rhev.lab.eng.brq.redhat.com:/var/lib/exports/iso returned by VDSM was: Problem while trying to mount target >2014-07-30 15:36:09,534 ERROR [org.ovirt.engine.core.bll.storage.NFSStorageHelper] (org.ovirt.thread.pool-8-thread-20) [26e7de54] The connection with details pb-rh35.rhev.lab.eng.brq.redhat.com:/var/lib/exports/iso failed because of error code 477 and error message is: problem while trying to mount target >2014-07-30 15:36:09,538 ERROR [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-8-thread-20) [26e7de54] Transaction rolled-back for command: org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand. >2014-07-30 15:36:09,540 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.AttachStorageDomainVDSCommand] (org.ovirt.thread.pool-8-thread-29) [4b747578] START, AttachStorageDomainVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, storageDomainId = 63fe3516-4170-402a-9dc8-6d3eab1c7933), log id: 26013a9b >2014-07-30 15:38:09,967 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.AttachStorageDomainVDSCommand] (org.ovirt.thread.pool-8-thread-29) [4b747578] Failed in AttachStorageDomainVDS method >2014-07-30 15:38:10,000 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (org.ovirt.thread.pool-8-thread-29) [4b747578] IrsBroker::Failed::AttachStorageDomainVDS due to: IRSErrorException: IRSGenericException: IRSErrorException: Failed to AttachStorageDomainVDS, error = Storage domain does not exist: ('63fe3516-4170-402a-9dc8-6d3eab1c7933',), code = 358 >2014-07-30 15:38:10,010 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.AttachStorageDomainVDSCommand] (org.ovirt.thread.pool-8-thread-29) [4b747578] FINISH, AttachStorageDomainVDSCommand, log id: 26013a9b >2014-07-30 15:38:10,011 ERROR [org.ovirt.engine.core.bll.storage.AttachStorageDomainToPoolCommand] (org.ovirt.thread.pool-8-thread-29) [4b747578] Command org.ovirt.engine.core.bll.storage.AttachStorageDomainToPoolCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.irsbroker.IRSErrorException: IRSGenericException: IRSErrorException: Failed to AttachStorageDomainVDS, error = Storage domain does not exist: ('63fe3516-4170-402a-9dc8-6d3eab1c7933',), code = 358 (Failed with error StorageDomainDoesNotExist and code 358) >2014-07-30 15:38:10,017 INFO [org.ovirt.engine.core.bll.storage.AttachStorageDomainToPoolCommand] (org.ovirt.thread.pool-8-thread-29) [4b747578] Command [id=7d68ad4e-6f96-4ed6-8fff-9a5456a4cad2]: Compensating NEW_ENTITY_ID of org.ovirt.engine.core.common.businessentities.StoragePoolIsoMap; snapshot: storagePoolId = 00000002-0002-0002-0002-0000000001d9, storageId = 63fe3516-4170-402a-9dc8-6d3eab1c7933. >2014-07-30 15:38:10,029 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-29) [4b747578] Correlation ID: 4b747578, Job ID: d2fd16d1-43cc-4ff2-9e72-cae105b93986, Call Stack: null, Custom Event ID: -1, Message: Failed to attach Storage Domain ISO_DOMAIN to Data Center Default. (User: admin) >2014-07-30 15:43:24,401 INFO [org.ovirt.engine.core.bll.storage.AddStorageServerConnectionCommand] (ajp--127.0.0.1-8702-3) [40864998] Lock Acquired to object EngineLock [exclusiveLocks= key: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35export value: STORAGE_CONNECTION >, sharedLocks= ] >2014-07-30 15:43:24,412 INFO [org.ovirt.engine.core.bll.storage.AddStorageServerConnectionCommand] (ajp--127.0.0.1-8702-3) [40864998] Running command: AddStorageServerConnectionCommand internal: false. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2014-07-30 15:43:24,415 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (ajp--127.0.0.1-8702-3) [40864998] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: null, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35export, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 4bdcd55 >2014-07-30 15:43:25,107 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (ajp--127.0.0.1-8702-3) [40864998] FINISH, ConnectStorageServerVDSCommand, return: {00000000-0000-0000-0000-000000000000=0}, log id: 4bdcd55 >2014-07-30 15:43:25,115 INFO [org.ovirt.engine.core.bll.storage.AddStorageServerConnectionCommand] (ajp--127.0.0.1-8702-3) [40864998] Lock freed to object EngineLock [exclusiveLocks= key: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35export value: STORAGE_CONNECTION >, sharedLocks= ] >2014-07-30 15:43:25,220 INFO [org.ovirt.engine.core.bll.storage.AddNFSStorageDomainCommand] (ajp--127.0.0.1-8702-11) [43c29a81] Running command: AddNFSStorageDomainCommand internal: false. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2014-07-30 15:43:25,230 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (ajp--127.0.0.1-8702-11) [43c29a81] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: c532e2a4-87d2-4ee2-a1cc-ca6ec88171bb, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35export, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 75251f19 >2014-07-30 15:43:25,282 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (ajp--127.0.0.1-8702-11) [43c29a81] FINISH, ConnectStorageServerVDSCommand, return: {c532e2a4-87d2-4ee2-a1cc-ca6ec88171bb=0}, log id: 75251f19 >2014-07-30 15:43:25,285 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateStorageDomainVDSCommand] (ajp--127.0.0.1-8702-11) [43c29a81] START, CreateStorageDomainVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storageDomain=org.ovirt.engine.core.common.businessentities.StorageDomainStatic@a1a50139, args=10.34.63.202:/mnt/export/nfs/lv3/pbenas/35export), log id: 6b3a6865 >2014-07-30 15:44:26,213 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateStorageDomainVDSCommand] (ajp--127.0.0.1-8702-11) [43c29a81] FINISH, CreateStorageDomainVDSCommand, log id: 6b3a6865 >2014-07-30 15:44:26,215 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetStorageDomainStatsVDSCommand] (ajp--127.0.0.1-8702-11) [43c29a81] START, GetStorageDomainStatsVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storageDomainId=453fe8a4-ef9b-4952-bf43-756dc497291e), log id: 70991c2c >2014-07-30 15:44:26,266 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetStorageDomainStatsVDSCommand] (ajp--127.0.0.1-8702-11) [43c29a81] FINISH, GetStorageDomainStatsVDSCommand, return: org.ovirt.engine.core.common.businessentities.StorageDomain@5eab6273, log id: 70991c2c >2014-07-30 15:44:26,279 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-11) [43c29a81] Correlation ID: 43c29a81, Job ID: 497a0273-48d0-4630-bf2e-6b68a26ffa45, Call Stack: null, Custom Event ID: -1, Message: Storage Domain export was added by admin >2014-07-30 15:44:26,679 INFO [org.ovirt.engine.core.bll.storage.DisconnectStorageServerConnectionCommand] (ajp--127.0.0.1-8702-1) [7147f2dc] Running command: DisconnectStorageServerConnectionCommand internal: false. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2014-07-30 15:44:26,682 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (ajp--127.0.0.1-8702-1) [7147f2dc] START, DisconnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: c532e2a4-87d2-4ee2-a1cc-ca6ec88171bb, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35export, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 716ea070 >2014-07-30 15:44:26,884 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (ajp--127.0.0.1-8702-1) [7147f2dc] FINISH, DisconnectStorageServerVDSCommand, return: {c532e2a4-87d2-4ee2-a1cc-ca6ec88171bb=0}, log id: 716ea070 >2014-07-30 15:44:27,347 INFO [org.ovirt.engine.core.bll.storage.AttachStorageDomainToPoolCommand] (ajp--127.0.0.1-8702-6) [7c810b8b] Running command: AttachStorageDomainToPoolCommand internal: false. Entities affected : ID: 453fe8a4-ef9b-4952-bf43-756dc497291e Type: Storage >2014-07-30 15:44:27,357 INFO [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-8-thread-11) [6834c80] Running command: ConnectStorageToVdsCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2014-07-30 15:44:27,361 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-11) [6834c80] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: c532e2a4-87d2-4ee2-a1cc-ca6ec88171bb, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35export, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 5254a13d >2014-07-30 15:44:27,403 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-11) [6834c80] FINISH, ConnectStorageServerVDSCommand, return: {c532e2a4-87d2-4ee2-a1cc-ca6ec88171bb=0}, log id: 5254a13d >2014-07-30 15:44:27,407 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetStorageDomainInfoVDSCommand] (ajp--127.0.0.1-8702-6) [7c810b8b] START, HSMGetStorageDomainInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storageDomainId=453fe8a4-ef9b-4952-bf43-756dc497291e), log id: 49cce2a3 >2014-07-30 15:44:27,630 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetStorageDomainInfoVDSCommand] (ajp--127.0.0.1-8702-6) [7c810b8b] FINISH, HSMGetStorageDomainInfoVDSCommand, return: <org.ovirt.engine.core.common.businessentities.StorageDomainStatic@b31ea5b3, null>, log id: 49cce2a3 >2014-07-30 15:44:27,632 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.AttachStorageDomainVDSCommand] (ajp--127.0.0.1-8702-6) [7c810b8b] START, AttachStorageDomainVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, storageDomainId = 453fe8a4-ef9b-4952-bf43-756dc497291e), log id: 2adbe1a2 >2014-07-30 15:44:51,784 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.AttachStorageDomainVDSCommand] (ajp--127.0.0.1-8702-6) [7c810b8b] FINISH, AttachStorageDomainVDSCommand, log id: 2adbe1a2 >2014-07-30 15:44:51,821 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetImagesListVDSCommand] (ajp--127.0.0.1-8702-6) [7c810b8b] START, GetImagesListVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, sdUUID = 453fe8a4-ef9b-4952-bf43-756dc497291e), log id: 594c9668 >2014-07-30 15:44:51,887 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetImagesListVDSCommand] (ajp--127.0.0.1-8702-6) [7c810b8b] FINISH, GetImagesListVDSCommand, return: [], log id: 594c9668 >2014-07-30 15:44:51,894 WARN [org.ovirt.engine.core.bll.storage.AttachStorageDomainToPoolCommand] (ajp--127.0.0.1-8702-6) There are no OVF_STORE disks on storage domain id 453fe8a4-ef9b-4952-bf43-756dc497291e >2014-07-30 15:44:51,900 INFO [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (ajp--127.0.0.1-8702-6) [112350cc] Lock Acquired to object EngineLock [exclusiveLocks= key: 453fe8a4-ef9b-4952-bf43-756dc497291e value: STORAGE >, sharedLocks= ] >2014-07-30 15:44:51,911 INFO [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (ajp--127.0.0.1-8702-6) [112350cc] Running command: ActivateStorageDomainCommand internal: true. Entities affected : ID: 453fe8a4-ef9b-4952-bf43-756dc497291e Type: Storage >2014-07-30 15:44:51,916 INFO [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (ajp--127.0.0.1-8702-6) [112350cc] Lock freed to object EngineLock [exclusiveLocks= key: 453fe8a4-ef9b-4952-bf43-756dc497291e value: STORAGE >, sharedLocks= ] >2014-07-30 15:44:51,917 INFO [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (ajp--127.0.0.1-8702-6) [112350cc] ActivateStorage Domain. Before Connect all hosts to pool. Time:7/30/14 3:44 PM >2014-07-30 15:44:51,921 INFO [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-8-thread-3) [2aa6df37] Running command: ConnectStorageToVdsCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2014-07-30 15:44:51,924 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-3) [2aa6df37] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: c532e2a4-87d2-4ee2-a1cc-ca6ec88171bb, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35export, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 5e8de28a >2014-07-30 15:44:51,938 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-3) [2aa6df37] FINISH, ConnectStorageServerVDSCommand, return: {c532e2a4-87d2-4ee2-a1cc-ca6ec88171bb=0}, log id: 5e8de28a >2014-07-30 15:44:51,940 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ActivateStorageDomainVDSCommand] (ajp--127.0.0.1-8702-6) [112350cc] START, ActivateStorageDomainVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, storageDomainId = 453fe8a4-ef9b-4952-bf43-756dc497291e), log id: 47005d16 >2014-07-30 15:44:52,174 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ActivateStorageDomainVDSCommand] (ajp--127.0.0.1-8702-6) [112350cc] FINISH, ActivateStorageDomainVDSCommand, log id: 47005d16 >2014-07-30 15:44:52,175 INFO [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (ajp--127.0.0.1-8702-6) [112350cc] ActivateStorage Domain. After Connect all hosts to pool. Time:7/30/14 3:44 PM >2014-07-30 15:44:52,186 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-8) START, ConnectStoragePoolVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, masterVersion = 1), log id: 2bc61ac >2014-07-30 15:45:52,429 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-8) FINISH, ConnectStoragePoolVDSCommand, log id: 2bc61ac >2014-07-30 15:45:52,430 INFO [org.ovirt.engine.core.bll.storage.RefreshPoolSingleAsyncOperation] (org.ovirt.thread.pool-8-thread-8) Refreshed vds rack in pool Default >2014-07-30 15:45:52,431 INFO [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (ajp--127.0.0.1-8702-6) [112350cc] ActivateStorage Domain. After change storage pool status in vds. Time:7/30/14 3:45 PM >2014-07-30 15:45:52,437 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) [112350cc] Correlation ID: 112350cc, Call Stack: null, Custom Event ID: -1, Message: Storage Domain export (Data Center Default) was activated by Non interactive user >2014-07-30 15:45:52,443 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) [112350cc] Correlation ID: 7c810b8b, Job ID: 98fe282b-4697-43b5-89a9-b3a389815624, Call Stack: null, Custom Event ID: -1, Message: Storage Domain export was attached to Data Center Default by admin >2014-07-30 16:01:49,627 ERROR [org.ovirt.engine.core.aaa.filters.BasicAuthenticationFilter] (ajp--127.0.0.1-8702-2) Error in obtaining profile null >2014-07-30 16:02:06,951 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp--127.0.0.1-8702-7) Running command: LoginUserCommand internal: false. >2014-07-30 16:02:06,958 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-7) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-07-30 16:02:06,975 ERROR [org.ovirt.engine.core.aaa.filters.RestApiSessionMgmtFilter] (ajp--127.0.0.1-8702-7) Session-TTL header was not passed. Not setting TTL value >2014-07-30 16:02:07,063 ERROR [org.ovirt.engine.core.aaa.filters.RestApiSessionMgmtFilter] (ajp--127.0.0.1-8702-10) Session-TTL header was not passed. Not setting TTL value >2014-07-30 16:02:07,841 ERROR [org.ovirt.engine.core.aaa.filters.RestApiSessionMgmtFilter] (ajp--127.0.0.1-8702-11) Session-TTL header was not passed. Not setting TTL value >2014-07-30 16:02:29,370 INFO [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (ajp--127.0.0.1-8702-1) [1d31befe] Lock Acquired to object EngineLock [exclusiveLocks= key: 453fe8a4-ef9b-4952-bf43-756dc497291e value: STORAGE >, sharedLocks= key: 00000002-0002-0002-0002-0000000001d9 value: POOL >] >2014-07-30 16:02:29,393 INFO [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-24) [1d31befe] Running command: DeactivateStorageDomainCommand internal: false. Entities affected : ID: 453fe8a4-ef9b-4952-bf43-756dc497291e Type: Storage >2014-07-30 16:02:29,400 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.DeactivateStorageDomainVDSCommand] (org.ovirt.thread.pool-8-thread-24) [1d31befe] START, DeactivateStorageDomainVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, storageDomainId = 453fe8a4-ef9b-4952-bf43-756dc497291e, masterDomainId = 00000000-0000-0000-0000-000000000000, masterVersion = 1), log id: 17d57ccd >2014-07-30 16:02:32,414 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.DeactivateStorageDomainVDSCommand] (org.ovirt.thread.pool-8-thread-24) [1d31befe] FINISH, DeactivateStorageDomainVDSCommand, log id: 17d57ccd >2014-07-30 16:02:32,416 INFO [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-24) [1d31befe] Lock freed to object EngineLock [exclusiveLocks= key: 453fe8a4-ef9b-4952-bf43-756dc497291e value: STORAGE >, sharedLocks= key: 00000002-0002-0002-0002-0000000001d9 value: POOL >] >2014-07-30 16:02:32,426 INFO [org.ovirt.engine.core.bll.storage.AfterDeactivateSingleAsyncOperation] (org.ovirt.thread.pool-8-thread-46) After deactivate treatment vds: rack,pool Default >2014-07-30 16:02:32,429 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-46) START, ConnectStoragePoolVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, masterVersion = 1), log id: 254c628d >2014-07-30 16:03:32,731 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-46) FINISH, ConnectStoragePoolVDSCommand, log id: 254c628d >2014-07-30 16:03:32,739 INFO [org.ovirt.engine.core.bll.storage.DisconnectStorageServerConnectionCommand] (org.ovirt.thread.pool-8-thread-24) [72fb412e] Running command: DisconnectStorageServerConnectionCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2014-07-30 16:03:32,742 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-24) [72fb412e] START, DisconnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: c532e2a4-87d2-4ee2-a1cc-ca6ec88171bb, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35export, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 16e0a177 >2014-07-30 16:03:32,795 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-12) Adding domain 453fe8a4-ef9b-4952-bf43-756dc497291e to the domains in maintenance cache >2014-07-30 16:03:32,799 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-21) Moving domain 453fe8a4-ef9b-4952-bf43-756dc497291e to maintenance >2014-07-30 16:03:32,805 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-9) Moving domain 453fe8a4-ef9b-4952-bf43-756dc497291e to maintenance >2014-07-30 16:03:32,811 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-20) Moving domain 453fe8a4-ef9b-4952-bf43-756dc497291e to maintenance >2014-07-30 16:03:32,816 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-38) Moving domain 453fe8a4-ef9b-4952-bf43-756dc497291e to maintenance >2014-07-30 16:03:32,944 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-24) [72fb412e] FINISH, DisconnectStorageServerVDSCommand, return: {c532e2a4-87d2-4ee2-a1cc-ca6ec88171bb=0}, log id: 16e0a177 >2014-07-30 16:03:32,947 INFO [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-24) [72fb412e] Domain 453fe8a4-ef9b-4952-bf43-756dc497291e will remain in PreparingForMaintenance status until deactivated on all hosts >2014-07-30 16:03:32,956 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-24) [72fb412e] Correlation ID: 1d31befe, Job ID: 251985e0-17b4-4d35-ad28-6754cc9c1449, Call Stack: null, Custom Event ID: -1, Message: Storage Domain export (Data Center Default) was deactivated by admin >2014-07-30 16:03:34,069 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-35) Moving domain 453fe8a4-ef9b-4952-bf43-756dc497291e to maintenance >2014-07-30 16:05:05,843 INFO [org.ovirt.engine.core.bll.storage.DetachStorageDomainFromPoolCommand] (org.ovirt.thread.pool-8-thread-24) [675e19b6] Running command: DetachStorageDomainFromPoolCommand internal: false. Entities affected : ID: 453fe8a4-ef9b-4952-bf43-756dc497291e Type: Storage >2014-07-30 16:05:05,845 INFO [org.ovirt.engine.core.bll.storage.DetachStorageDomainFromPoolCommand] (org.ovirt.thread.pool-8-thread-24) [675e19b6] Start detach storage domain >2014-07-30 16:05:05,849 INFO [org.ovirt.engine.core.bll.storage.DetachStorageDomainFromPoolCommand] (org.ovirt.thread.pool-8-thread-24) [675e19b6] Detach storage domain: before connect >2014-07-30 16:05:05,854 INFO [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-8-thread-18) [57fef3b0] Running command: ConnectStorageToVdsCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2014-07-30 16:05:05,859 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-18) [57fef3b0] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: c532e2a4-87d2-4ee2-a1cc-ca6ec88171bb, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35export, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 33c2e97 >2014-07-30 16:05:05,916 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-18) [57fef3b0] FINISH, ConnectStorageServerVDSCommand, return: {c532e2a4-87d2-4ee2-a1cc-ca6ec88171bb=0}, log id: 33c2e97 >2014-07-30 16:05:05,919 INFO [org.ovirt.engine.core.bll.storage.DetachStorageDomainFromPoolCommand] (org.ovirt.thread.pool-8-thread-24) [675e19b6] Detach storage domain: after connect >2014-07-30 16:05:05,920 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.DetachStorageDomainVDSCommand] (org.ovirt.thread.pool-8-thread-24) [675e19b6] START, DetachStorageDomainVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, storageDomainId = 453fe8a4-ef9b-4952-bf43-756dc497291e, masterDomainId = 00000000-0000-0000-0000-000000000000, masterVersion = 1, force = false), log id: 769677f2 >2014-07-30 16:05:06,107 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.DetachStorageDomainVDSCommand] (org.ovirt.thread.pool-8-thread-24) [675e19b6] FINISH, DetachStorageDomainVDSCommand, log id: 769677f2 >2014-07-30 16:05:06,108 INFO [org.ovirt.engine.core.bll.storage.DetachStorageDomainFromPoolCommand] (org.ovirt.thread.pool-8-thread-24) [675e19b6] Detach storage domain: after detach in vds >2014-07-30 16:05:06,117 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-45) START, ConnectStoragePoolVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, masterVersion = 1), log id: 5a072551 >2014-07-30 16:06:06,419 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-45) FINISH, ConnectStoragePoolVDSCommand, log id: 5a072551 >2014-07-30 16:06:06,422 INFO [org.ovirt.engine.core.bll.storage.DisconnectStorageServerConnectionCommand] (org.ovirt.thread.pool-8-thread-45) [36a172] Running command: DisconnectStorageServerConnectionCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2014-07-30 16:06:06,426 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-45) [36a172] START, DisconnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: c532e2a4-87d2-4ee2-a1cc-ca6ec88171bb, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35export, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 2954ffef >2014-07-30 16:06:06,619 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-45) [36a172] FINISH, DisconnectStorageServerVDSCommand, return: {c532e2a4-87d2-4ee2-a1cc-ca6ec88171bb=0}, log id: 2954ffef >2014-07-30 16:06:06,622 INFO [org.ovirt.engine.core.bll.storage.DetachStorageDomainFromPoolCommand] (org.ovirt.thread.pool-8-thread-24) [675e19b6] Detach storage domain: after disconnect storage >2014-07-30 16:06:06,623 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-26) Removing domain 453fe8a4-ef9b-4952-bf43-756dc497291e from the domains in maintenance cache >2014-07-30 16:06:06,635 INFO [org.ovirt.engine.core.bll.storage.DetachStorageDomainFromPoolCommand] (org.ovirt.thread.pool-8-thread-24) [675e19b6] End detach storage domain >2014-07-30 16:06:06,641 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-24) [675e19b6] Correlation ID: 675e19b6, Job ID: 227d28da-8129-41c9-873f-f1451ed7d3eb, Call Stack: null, Custom Event ID: -1, Message: Storage Domain export was detached from Data Center Default by admin >2014-07-30 16:06:09,278 INFO [org.ovirt.engine.core.bll.storage.RemoveStorageDomainCommand] (ajp--127.0.0.1-8702-2) [226334ea] Lock Acquired to object EngineLock [exclusiveLocks= key: 453fe8a4-ef9b-4952-bf43-756dc497291e value: STORAGE >, sharedLocks= ] >2014-07-30 16:06:09,288 INFO [org.ovirt.engine.core.bll.storage.RemoveStorageDomainCommand] (ajp--127.0.0.1-8702-2) [226334ea] Running command: RemoveStorageDomainCommand internal: false. Entities affected : ID: 453fe8a4-ef9b-4952-bf43-756dc497291e Type: Storage >2014-07-30 16:06:09,291 INFO [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (ajp--127.0.0.1-8702-2) [674b6284] Running command: ConnectStorageToVdsCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2014-07-30 16:06:09,294 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (ajp--127.0.0.1-8702-2) [674b6284] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: c532e2a4-87d2-4ee2-a1cc-ca6ec88171bb, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35export, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 7a008e50 >2014-07-30 16:06:09,340 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (ajp--127.0.0.1-8702-2) [674b6284] FINISH, ConnectStorageServerVDSCommand, return: {c532e2a4-87d2-4ee2-a1cc-ca6ec88171bb=0}, log id: 7a008e50 >2014-07-30 16:06:09,344 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FormatStorageDomainVDSCommand] (ajp--127.0.0.1-8702-2) [674b6284] START, FormatStorageDomainVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storageDomainId=453fe8a4-ef9b-4952-bf43-756dc497291e), log id: 55dc326c >2014-07-30 16:06:09,806 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FormatStorageDomainVDSCommand] (ajp--127.0.0.1-8702-2) [674b6284] FINISH, FormatStorageDomainVDSCommand, log id: 55dc326c >2014-07-30 16:06:09,809 INFO [org.ovirt.engine.core.bll.storage.DisconnectStorageServerConnectionCommand] (ajp--127.0.0.1-8702-2) [10e97f08] Running command: DisconnectStorageServerConnectionCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2014-07-30 16:06:09,811 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (ajp--127.0.0.1-8702-2) [10e97f08] START, DisconnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: c532e2a4-87d2-4ee2-a1cc-ca6ec88171bb, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35export, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 581e493 >2014-07-30 16:06:09,991 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (ajp--127.0.0.1-8702-2) [10e97f08] FINISH, DisconnectStorageServerVDSCommand, return: {c532e2a4-87d2-4ee2-a1cc-ca6ec88171bb=0}, log id: 581e493 >2014-07-30 16:06:10,068 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-2) [10e97f08] Correlation ID: 226334ea, Job ID: f1e8c884-9ab1-44e7-8c72-d30fcab21078, Call Stack: null, Custom Event ID: -1, Message: Storage Domain export was removed by admin >2014-07-30 16:06:10,074 INFO [org.ovirt.engine.core.bll.storage.RemoveStorageDomainCommand] (ajp--127.0.0.1-8702-2) [10e97f08] Lock freed to object EngineLock [exclusiveLocks= key: 453fe8a4-ef9b-4952-bf43-756dc497291e value: STORAGE >, sharedLocks= ] >2014-07-30 16:06:56,566 INFO [org.ovirt.engine.core.bll.storage.AddStorageServerConnectionCommand] (ajp--127.0.0.1-8702-11) [387f6226] Lock Acquired to object EngineLock [exclusiveLocks= key: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35export value: STORAGE_CONNECTION >, sharedLocks= ] >2014-07-30 16:06:56,580 INFO [org.ovirt.engine.core.bll.storage.AddStorageServerConnectionCommand] (ajp--127.0.0.1-8702-11) [387f6226] Running command: AddStorageServerConnectionCommand internal: false. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2014-07-30 16:06:56,582 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (ajp--127.0.0.1-8702-11) [387f6226] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: null, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35export, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 4320772c >2014-07-30 16:06:56,624 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (ajp--127.0.0.1-8702-11) [387f6226] FINISH, ConnectStorageServerVDSCommand, return: {00000000-0000-0000-0000-000000000000=0}, log id: 4320772c >2014-07-30 16:06:56,631 INFO [org.ovirt.engine.core.bll.storage.AddStorageServerConnectionCommand] (ajp--127.0.0.1-8702-11) [387f6226] Lock freed to object EngineLock [exclusiveLocks= key: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35export value: STORAGE_CONNECTION >, sharedLocks= ] >2014-07-30 16:06:56,754 INFO [org.ovirt.engine.core.bll.storage.AddNFSStorageDomainCommand] (ajp--127.0.0.1-8702-10) [24d39fbe] Running command: AddNFSStorageDomainCommand internal: false. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2014-07-30 16:06:56,766 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (ajp--127.0.0.1-8702-10) [24d39fbe] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: 05047c9a-152e-4ef2-b118-4a1753c1eae0, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35export, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 28834bf6 >2014-07-30 16:06:56,780 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (ajp--127.0.0.1-8702-10) [24d39fbe] FINISH, ConnectStorageServerVDSCommand, return: {05047c9a-152e-4ef2-b118-4a1753c1eae0=0}, log id: 28834bf6 >2014-07-30 16:06:56,783 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateStorageDomainVDSCommand] (ajp--127.0.0.1-8702-10) [24d39fbe] START, CreateStorageDomainVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storageDomain=org.ovirt.engine.core.common.businessentities.StorageDomainStatic@3a6e99d4, args=10.34.63.202:/mnt/export/nfs/lv3/pbenas/35export), log id: 6623ab32 >2014-07-30 16:07:57,529 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateStorageDomainVDSCommand] (ajp--127.0.0.1-8702-10) [24d39fbe] FINISH, CreateStorageDomainVDSCommand, log id: 6623ab32 >2014-07-30 16:07:57,533 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetStorageDomainStatsVDSCommand] (ajp--127.0.0.1-8702-10) [24d39fbe] START, GetStorageDomainStatsVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storageDomainId=44559ec0-6bef-491c-bbd6-e67278f37987), log id: 19da4130 >2014-07-30 16:07:57,586 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetStorageDomainStatsVDSCommand] (ajp--127.0.0.1-8702-10) [24d39fbe] FINISH, GetStorageDomainStatsVDSCommand, return: org.ovirt.engine.core.common.businessentities.StorageDomain@9c054809, log id: 19da4130 >2014-07-30 16:07:57,599 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-10) [24d39fbe] Correlation ID: 24d39fbe, Job ID: 151e336d-46e4-426a-b84e-73c9286efb3a, Call Stack: null, Custom Event ID: -1, Message: Storage Domain export was added by admin >2014-07-30 16:07:57,714 INFO [org.ovirt.engine.core.bll.storage.DisconnectStorageServerConnectionCommand] (ajp--127.0.0.1-8702-7) [4da90475] Running command: DisconnectStorageServerConnectionCommand internal: false. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2014-07-30 16:07:57,721 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (ajp--127.0.0.1-8702-7) [4da90475] START, DisconnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: 05047c9a-152e-4ef2-b118-4a1753c1eae0, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35export, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 23079a1a >2014-07-30 16:07:57,895 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (ajp--127.0.0.1-8702-7) [4da90475] FINISH, DisconnectStorageServerVDSCommand, return: {05047c9a-152e-4ef2-b118-4a1753c1eae0=0}, log id: 23079a1a >2014-07-30 16:07:58,110 INFO [org.ovirt.engine.core.bll.storage.AttachStorageDomainToPoolCommand] (ajp--127.0.0.1-8702-11) [62268b88] Running command: AttachStorageDomainToPoolCommand internal: false. Entities affected : ID: 44559ec0-6bef-491c-bbd6-e67278f37987 Type: Storage >2014-07-30 16:07:58,121 INFO [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-8-thread-48) [5d6494d6] Running command: ConnectStorageToVdsCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2014-07-30 16:07:58,127 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-48) [5d6494d6] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: 05047c9a-152e-4ef2-b118-4a1753c1eae0, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35export, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 5a304423 >2014-07-30 16:07:58,171 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-48) [5d6494d6] FINISH, ConnectStorageServerVDSCommand, return: {05047c9a-152e-4ef2-b118-4a1753c1eae0=0}, log id: 5a304423 >2014-07-30 16:07:58,173 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.AttachStorageDomainVDSCommand] (ajp--127.0.0.1-8702-11) [62268b88] START, AttachStorageDomainVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, storageDomainId = 44559ec0-6bef-491c-bbd6-e67278f37987), log id: 214d7abf >2014-07-30 16:08:19,951 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.AttachStorageDomainVDSCommand] (ajp--127.0.0.1-8702-11) [62268b88] FINISH, AttachStorageDomainVDSCommand, log id: 214d7abf >2014-07-30 16:08:19,985 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetImagesListVDSCommand] (ajp--127.0.0.1-8702-11) [62268b88] START, GetImagesListVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, sdUUID = 44559ec0-6bef-491c-bbd6-e67278f37987), log id: 26400e66 >2014-07-30 16:08:20,050 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetImagesListVDSCommand] (ajp--127.0.0.1-8702-11) [62268b88] FINISH, GetImagesListVDSCommand, return: [], log id: 26400e66 >2014-07-30 16:08:20,057 WARN [org.ovirt.engine.core.bll.storage.AttachStorageDomainToPoolCommand] (ajp--127.0.0.1-8702-11) There are no OVF_STORE disks on storage domain id 44559ec0-6bef-491c-bbd6-e67278f37987 >2014-07-30 16:08:20,065 INFO [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (ajp--127.0.0.1-8702-11) [3711efe3] Lock Acquired to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE >, sharedLocks= ] >2014-07-30 16:08:20,079 INFO [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (ajp--127.0.0.1-8702-11) [3711efe3] Running command: ActivateStorageDomainCommand internal: true. Entities affected : ID: 44559ec0-6bef-491c-bbd6-e67278f37987 Type: Storage >2014-07-30 16:08:20,086 INFO [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (ajp--127.0.0.1-8702-11) [3711efe3] Lock freed to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE >, sharedLocks= ] >2014-07-30 16:08:20,087 INFO [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (ajp--127.0.0.1-8702-11) [3711efe3] ActivateStorage Domain. Before Connect all hosts to pool. Time:7/30/14 4:08 PM >2014-07-30 16:08:20,092 INFO [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-8-thread-45) [7d791d77] Running command: ConnectStorageToVdsCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2014-07-30 16:08:20,095 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-45) [7d791d77] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: 05047c9a-152e-4ef2-b118-4a1753c1eae0, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35export, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 4e311b55 >2014-07-30 16:08:20,110 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-45) [7d791d77] FINISH, ConnectStorageServerVDSCommand, return: {05047c9a-152e-4ef2-b118-4a1753c1eae0=0}, log id: 4e311b55 >2014-07-30 16:08:20,112 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ActivateStorageDomainVDSCommand] (ajp--127.0.0.1-8702-11) [3711efe3] START, ActivateStorageDomainVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, storageDomainId = 44559ec0-6bef-491c-bbd6-e67278f37987), log id: 17bc6586 >2014-07-30 16:08:20,349 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ActivateStorageDomainVDSCommand] (ajp--127.0.0.1-8702-11) [3711efe3] FINISH, ActivateStorageDomainVDSCommand, log id: 17bc6586 >2014-07-30 16:08:20,350 INFO [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (ajp--127.0.0.1-8702-11) [3711efe3] ActivateStorage Domain. After Connect all hosts to pool. Time:7/30/14 4:08 PM >2014-07-30 16:08:20,361 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-8) START, ConnectStoragePoolVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, masterVersion = 1), log id: c855534 >2014-07-30 16:09:04,456 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp--127.0.0.1-8702-5) Running command: LoginUserCommand internal: false. >2014-07-30 16:09:04,464 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-5) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-07-30 16:09:04,465 ERROR [org.ovirt.engine.core.aaa.filters.RestApiSessionMgmtFilter] (ajp--127.0.0.1-8702-5) Session-TTL header was not passed. Not setting TTL value >2014-07-30 16:09:04,552 ERROR [org.ovirt.engine.core.aaa.filters.RestApiSessionMgmtFilter] (ajp--127.0.0.1-8702-4) Session-TTL header was not passed. Not setting TTL value >2014-07-30 16:09:04,580 ERROR [org.ovirt.engine.core.aaa.filters.RestApiSessionMgmtFilter] (ajp--127.0.0.1-8702-3) Session-TTL header was not passed. Not setting TTL value >2014-07-30 16:09:20,656 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-8) FINISH, ConnectStoragePoolVDSCommand, log id: c855534 >2014-07-30 16:09:20,658 INFO [org.ovirt.engine.core.bll.storage.RefreshPoolSingleAsyncOperation] (org.ovirt.thread.pool-8-thread-8) Refreshed vds rack in pool Default >2014-07-30 16:09:20,670 INFO [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (ajp--127.0.0.1-8702-11) [3711efe3] ActivateStorage Domain. After change storage pool status in vds. Time:7/30/14 4:09 PM >2014-07-30 16:09:20,677 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-11) [3711efe3] Correlation ID: 3711efe3, Call Stack: null, Custom Event ID: -1, Message: Storage Domain export (Data Center Default) was activated by Non interactive user >2014-07-30 16:09:20,688 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-11) [3711efe3] Correlation ID: 62268b88, Job ID: 83c9e166-e27f-42ff-a514-5cc368113dc2, Call Stack: null, Custom Event ID: -1, Message: Storage Domain export was attached to Data Center Default by admin >2014-07-30 16:09:43,243 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp--127.0.0.1-8702-7) Running command: LoginUserCommand internal: false. >2014-07-30 16:09:43,251 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-7) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-07-30 16:09:43,254 ERROR [org.ovirt.engine.core.aaa.filters.RestApiSessionMgmtFilter] (ajp--127.0.0.1-8702-7) Session-TTL header was not passed. Not setting TTL value >2014-07-30 16:09:43,352 ERROR [org.ovirt.engine.core.aaa.filters.RestApiSessionMgmtFilter] (ajp--127.0.0.1-8702-10) Session-TTL header was not passed. Not setting TTL value >2014-07-30 16:11:17,551 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp--127.0.0.1-8702-7) Running command: LoginUserCommand internal: false. >2014-07-30 16:11:17,555 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-7) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-07-30 16:11:17,556 ERROR [org.ovirt.engine.core.aaa.filters.RestApiSessionMgmtFilter] (ajp--127.0.0.1-8702-7) Session-TTL header was not passed. Not setting TTL value >2014-07-30 16:11:17,727 ERROR [org.ovirt.engine.core.aaa.filters.RestApiSessionMgmtFilter] (ajp--127.0.0.1-8702-10) Session-TTL header was not passed. Not setting TTL value >2014-07-30 16:13:48,310 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetVmsInfoVDSCommand] (ajp--127.0.0.1-8702-4) START, GetVmsInfoVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, storageDomainId = 44559ec0-6bef-491c-bbd6-e67278f37987, vmIdList = null), log id: 5994058d >2014-07-30 16:13:48,323 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetVmsInfoVDSCommand] (ajp--127.0.0.1-8702-4) FINISH, GetVmsInfoVDSCommand, log id: 5994058d >2014-07-30 16:13:55,817 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetVmsInfoVDSCommand] (ajp--127.0.0.1-8702-1) START, GetVmsInfoVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, storageDomainId = 44559ec0-6bef-491c-bbd6-e67278f37987, vmIdList = null), log id: 4470d32e >2014-07-30 16:13:55,829 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetVmsInfoVDSCommand] (ajp--127.0.0.1-8702-1) FINISH, GetVmsInfoVDSCommand, log id: 4470d32e >2014-07-30 16:14:12,653 INFO [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (ajp--127.0.0.1-8702-3) [cac56be] Lock Acquired to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE >, sharedLocks= ] >2014-07-30 16:14:12,676 INFO [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-50) [cac56be] Running command: DeactivateStorageDomainCommand internal: false. Entities affected : ID: 44559ec0-6bef-491c-bbd6-e67278f37987 Type: Storage >2014-07-30 16:14:12,685 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.DeactivateStorageDomainVDSCommand] (org.ovirt.thread.pool-8-thread-50) [cac56be] START, DeactivateStorageDomainVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, storageDomainId = 44559ec0-6bef-491c-bbd6-e67278f37987, masterDomainId = 00000000-0000-0000-0000-000000000000, masterVersion = 1), log id: 299791ba >2014-07-30 16:14:12,697 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.DeactivateStorageDomainVDSCommand] (org.ovirt.thread.pool-8-thread-50) [cac56be] FINISH, DeactivateStorageDomainVDSCommand, log id: 299791ba >2014-07-30 16:14:12,698 INFO [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-50) [cac56be] Lock freed to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE >, sharedLocks= ] >2014-07-30 16:14:12,708 INFO [org.ovirt.engine.core.bll.storage.AfterDeactivateSingleAsyncOperation] (org.ovirt.thread.pool-8-thread-32) After deactivate treatment vds: rack,pool Default >2014-07-30 16:14:12,711 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-32) START, ConnectStoragePoolVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, masterVersion = 1), log id: 752d8055 >2014-07-30 16:15:12,961 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-32) FINISH, ConnectStoragePoolVDSCommand, log id: 752d8055 >2014-07-30 16:15:12,965 INFO [org.ovirt.engine.core.bll.storage.DisconnectStorageServerConnectionCommand] (org.ovirt.thread.pool-8-thread-50) [52520451] Running command: DisconnectStorageServerConnectionCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2014-07-30 16:15:12,970 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-22) Adding domain 44559ec0-6bef-491c-bbd6-e67278f37987 to the domains in maintenance cache >2014-07-30 16:15:12,974 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-35) Moving domain 44559ec0-6bef-491c-bbd6-e67278f37987 to maintenance >2014-07-30 16:15:12,975 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-50) [52520451] START, DisconnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: 05047c9a-152e-4ef2-b118-4a1753c1eae0, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35export, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 4aba71d2 >2014-07-30 16:15:12,980 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-18) Moving domain 44559ec0-6bef-491c-bbd6-e67278f37987 to maintenance >2014-07-30 16:15:12,986 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-39) Moving domain 44559ec0-6bef-491c-bbd6-e67278f37987 to maintenance >2014-07-30 16:15:12,991 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-29) Moving domain 44559ec0-6bef-491c-bbd6-e67278f37987 to maintenance >2014-07-30 16:15:12,996 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-16) Moving domain 44559ec0-6bef-491c-bbd6-e67278f37987 to maintenance >2014-07-30 16:15:13,166 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-50) [52520451] FINISH, DisconnectStorageServerVDSCommand, return: {05047c9a-152e-4ef2-b118-4a1753c1eae0=0}, log id: 4aba71d2 >2014-07-30 16:15:13,171 INFO [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-50) [52520451] Domain 44559ec0-6bef-491c-bbd6-e67278f37987 will remain in PreparingForMaintenance status until deactivated on all hosts >2014-07-30 16:15:13,181 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-50) [52520451] Correlation ID: cac56be, Job ID: 28d66c42-7ba2-42ff-afd7-eb605a369439, Call Stack: null, Custom Event ID: -1, Message: Storage Domain export (Data Center Default) was deactivated by admin >2014-07-30 16:15:22,737 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-37) Moving domain 44559ec0-6bef-491c-bbd6-e67278f37987 to maintenance >2014-07-30 16:17:13,328 INFO [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (ajp--127.0.0.1-8702-5) [6df02cad] Lock Acquired to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE >, sharedLocks= ] >2014-07-30 16:17:13,361 INFO [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-24) [6df02cad] Running command: ActivateStorageDomainCommand internal: false. Entities affected : ID: 44559ec0-6bef-491c-bbd6-e67278f37987 Type: Storage >2014-07-30 16:17:13,367 INFO [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-24) [6df02cad] Lock freed to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE >, sharedLocks= ] >2014-07-30 16:17:13,369 INFO [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-24) [6df02cad] ActivateStorage Domain. Before Connect all hosts to pool. Time:7/30/14 4:17 PM >2014-07-30 16:17:13,374 INFO [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-8-thread-45) [8cb39bb] Running command: ConnectStorageToVdsCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2014-07-30 16:17:13,378 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-45) [8cb39bb] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: 05047c9a-152e-4ef2-b118-4a1753c1eae0, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35export, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 7af25f08 >2014-07-30 16:17:13,437 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-45) [8cb39bb] FINISH, ConnectStorageServerVDSCommand, return: {05047c9a-152e-4ef2-b118-4a1753c1eae0=0}, log id: 7af25f08 >2014-07-30 16:17:13,439 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ActivateStorageDomainVDSCommand] (org.ovirt.thread.pool-8-thread-24) [6df02cad] START, ActivateStorageDomainVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, storageDomainId = 44559ec0-6bef-491c-bbd6-e67278f37987), log id: 7c841d9a >2014-07-30 16:17:13,676 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ActivateStorageDomainVDSCommand] (org.ovirt.thread.pool-8-thread-24) [6df02cad] FINISH, ActivateStorageDomainVDSCommand, log id: 7c841d9a >2014-07-30 16:17:13,677 INFO [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-24) [6df02cad] ActivateStorage Domain. After Connect all hosts to pool. Time:7/30/14 4:17 PM >2014-07-30 16:17:13,686 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-12) START, ConnectStoragePoolVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, masterVersion = 1), log id: 55f43dd4 >2014-07-30 16:18:13,950 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-12) FINISH, ConnectStoragePoolVDSCommand, log id: 55f43dd4 >2014-07-30 16:18:13,951 INFO [org.ovirt.engine.core.bll.storage.RefreshPoolSingleAsyncOperation] (org.ovirt.thread.pool-8-thread-12) Refreshed vds rack in pool Default >2014-07-30 16:18:13,952 INFO [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-24) [6df02cad] ActivateStorage Domain. After change storage pool status in vds. Time:7/30/14 4:18 PM >2014-07-30 16:18:13,953 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-15) Removing domain 44559ec0-6bef-491c-bbd6-e67278f37987 from the domains in maintenance cache >2014-07-30 16:18:13,961 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-24) [6df02cad] Correlation ID: 6df02cad, Job ID: d71d1d44-dea9-431e-af30-11724b0f4324, Call Stack: null, Custom Event ID: -1, Message: Storage Domain export (Data Center Default) was activated by admin >2014-07-30 16:18:29,070 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-40) domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 in problem. vds: rack >2014-07-30 16:18:29,072 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-40) domain 44559ec0-6bef-491c-bbd6-e67278f37987:export in problem. vds: rack >2014-07-30 16:18:29,073 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-40) Host 17f7fe56-5f32-40ef-b098-611b84484e0b has reported new storage access problem to the following domains 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752,44559ec0-6bef-491c-bbd6-e67278f37987 marking it for storage connections and pool metadata refresh (report id: e8a103cd-8125-46cd-bd80-a65117c56e2f) >2014-07-30 16:18:29,302 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-12) Running storage connections refresh for hosts [17f7fe56-5f32-40ef-b098-611b84484e0b] >2014-07-30 16:18:29,305 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (org.ovirt.thread.pool-8-thread-17) [4c3f40c7] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-07-30 16:18:29,316 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-17) [4c3f40c7] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, storageType = NFS, connectionList = [{ id: 05047c9a-152e-4ef2-b118-4a1753c1eae0, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35export, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 5cc65080 >2014-07-30 16:20:29,332 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-17) [4c3f40c7] FINISH, ConnectStorageServerVDSCommand, return: {5083f370-af09-457b-9b73-3440e8b65a07=0, 05047c9a-152e-4ef2-b118-4a1753c1eae0=0}, log id: 5cc65080 >2014-07-30 16:20:29,333 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (org.ovirt.thread.pool-8-thread-17) [4c3f40c7] Host rack storage connection was succeeded >2014-07-30 16:20:29,334 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-12) Submitting to the event queue pool refresh for hosts [17f7fe56-5f32-40ef-b098-611b84484e0b] >2014-07-30 16:20:29,336 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-12) Running storage pool metadata refresh for hosts {1} >2014-07-30 16:20:29,341 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-21) START, ConnectStoragePoolVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, masterVersion = 1), log id: 33b2f639 >2014-07-30 16:21:35,762 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-26) Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2014-07-30 16:21:35,770 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-26) [6bd9692] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-07-30 16:21:35,774 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-26) [6bd9692] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Active to Unknown, reason : null >2014-07-30 16:21:35,778 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-26) [6bd9692] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Active to Unknown, reason : null >2014-07-30 16:21:35,795 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-26) [6bd9692] Correlation ID: 6bd9692, Call Stack: null, Custom Event ID: -1, Message: Invalid status on Data Center Default. Setting Data Center status to Non Responsive (On host rack, Error: Cannot access master storage domain). >2014-07-30 16:21:55,083 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetVmsInfoVDSCommand] (ajp--127.0.0.1-8702-6) START, GetVmsInfoVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, storageDomainId = 44559ec0-6bef-491c-bbd6-e67278f37987, vmIdList = null), log id: 2ae6fcc0 >2014-07-30 16:22:35,809 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-21) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=00000002-0002-0002-0002-0000000001d9, msdUUID=71f82fcd-4fa6-4f56-b228-f4fa4bd4f752']] >2014-07-30 16:22:35,811 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-21) HostName = rack >2014-07-30 16:22:35,812 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-21) Command ConnectStoragePoolVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, masterVersion = 1) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=00000002-0002-0002-0002-0000000001d9, msdUUID=71f82fcd-4fa6-4f56-b228-f4fa4bd4f752' >2014-07-30 16:22:35,815 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-21) FINISH, ConnectStoragePoolVDSCommand, log id: 33b2f639 >2014-07-30 16:23:29,073 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-15) starting processDomainRecovery for domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 >2014-07-30 16:23:29,080 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-15) Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 was reported by all hosts in status UP as problematic. Not moving the domain to NonOperational because it is being reconstructed now. >2014-07-30 16:23:29,119 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-15) [266abdd1] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage >2014-07-30 16:23:29,151 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-15) [266abdd1] START, ResetIrsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, ignoreStopFailed = true), log id: 52b4dda9 >2014-07-30 16:23:29,155 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-15) [266abdd1] START, SpmStopVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9), log id: 1ceee97e >2014-07-30 16:23:29,161 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-15) [266abdd1] SpmStopVDSCommand::Stopping SPM on vds rack, pool id 00000002-0002-0002-0002-0000000001d9 >2014-07-30 16:23:32,340 INFO [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (ajp--127.0.0.1-8702-2) [72233a31] Lock Acquired to object EngineLock [exclusiveLocks= key: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 value: STORAGE >, sharedLocks= ] >2014-07-30 16:23:32,362 INFO [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-41) [72233a31] Running command: ActivateStorageDomainCommand internal: false. Entities affected : ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage >2014-07-30 16:23:32,367 INFO [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-41) [72233a31] Lock freed to object EngineLock [exclusiveLocks= key: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 value: STORAGE >, sharedLocks= ] >2014-07-30 16:23:32,368 INFO [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-41) [72233a31] ActivateStorage Domain. Before Connect all hosts to pool. Time:7/30/14 4:23 PM >2014-07-30 16:23:32,373 INFO [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-8-thread-34) [20f1fb14] Running command: ConnectStorageToVdsCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2014-07-30 16:23:32,377 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-34) [20f1fb14] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 25be2568 >2014-07-30 16:23:55,153 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-57) [cd4c3ce] Principal admin::internal synchronized >2014-07-30 16:24:22,116 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-100) [52a75f18] Attempting to update VMs/Templates Ovf. >2014-07-30 16:24:35,829 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-26) [6bd9692] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2014-07-30 16:24:35,831 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-26) [6bd9692] hostFromVds::selectedVds - rack, spmStatus returned null! >2014-07-30 16:24:35,833 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-26) [6bd9692] IrsBroker::Failed::GetStoragePoolInfoVDS >2014-07-30 16:24:35,834 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-26) [6bd9692] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2014-07-30 16:25:35,784 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-34) [20f1fb14] Command ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]) execution failed. Exception: VDSNetworkException: org.apache.commons.httpclient.NoHttpResponseException: The server 10.34.62.208 failed to respond >2014-07-30 16:25:35,785 WARN [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-15) [266abdd1] could not stop spm of pool 00000002-0002-0002-0002-0000000001d9 on vds 17f7fe56-5f32-40ef-b098-611b84484e0b - reason: org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcRunTimeException: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException >2014-07-30 16:25:35,797 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-15) [266abdd1] FINISH, SpmStopVDSCommand, log id: 1ceee97e >2014-07-30 16:25:35,798 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-34) [20f1fb14] FINISH, ConnectStorageServerVDSCommand, log id: 25be2568 >2014-07-30 16:25:35,800 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (ajp--127.0.0.1-8702-6) Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-07-30 16:25:35,803 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (ajp--127.0.0.1-8702-6) hostFromVds::selectedVds - rack, spmStatus returned null! >2014-07-30 16:25:35,799 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ListVDSCommand] (DefaultQuartzScheduler_Worker-71) Command ListVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-07-30 16:25:35,804 ERROR [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-8-thread-34) [20f1fb14] Command org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: org.apache.commons.httpclient.NoHttpResponseException: The server 10.34.62.208 failed to respond (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-07-30 16:25:35,806 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-71) Host rack is not responding. It will stay in Connecting state for a grace period of $80 seconds and after that an attempt to fence the host will be issued. >2014-07-30 16:25:35,814 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-15) [266abdd1] FINISH, ResetIrsVDSCommand, log id: 52b4dda9 >2014-07-30 16:25:35,815 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetVmsInfoVDSCommand] (ajp--127.0.0.1-8702-6) FINISH, GetVmsInfoVDSCommand, log id: 2ae6fcc0 >2014-07-30 16:25:35,821 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-71) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:406) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.fetchRunningVms(VdsUpdateRunTimeInfo.java:971) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVmStats(VdsUpdateRunTimeInfo.java:920) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:638) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:235) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) [:1.7.0_55] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_55] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_55] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_55] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_55] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_55] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_55] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_55] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy82.list(Unknown Source) > at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source) [:1.7.0_55] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_55] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_55] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_55] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_55] > >2014-07-30 16:25:35,832 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-98) [5aa74089] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-07-30 16:25:35,853 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-98) [5aa74089] hostFromVds::selectedVds - rack, spmStatus returned null! >2014-07-30 16:25:35,830 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to retrieve VM/Templates information from export domain export >2014-07-30 16:25:35,827 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-15) [266abdd1] Correlation ID: 266abdd1, Job ID: ad361002-0602-4453-adab-899c1a473485, Call Stack: null, Custom Event ID: -1, Message: Failed to Reconstruct Master Domain for Data Center Default. >2014-07-30 16:25:35,827 ERROR [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-8-thread-34) [20f1fb14] Transaction rolled-back for command: org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand. >2014-07-30 16:25:35,860 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ActivateStorageDomainVDSCommand] (org.ovirt.thread.pool-8-thread-41) [72233a31] START, ActivateStorageDomainVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, storageDomainId = 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752), log id: 2050ebd4 >2014-07-30 16:25:35,865 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-15) [266abdd1] Removing vds [] from the domain in maintenance cache >2014-07-30 16:25:35,865 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ActivateStorageDomainVDSCommand] (org.ovirt.thread.pool-8-thread-41) [72233a31] FINISH, ActivateStorageDomainVDSCommand, log id: 2050ebd4 >2014-07-30 16:25:35,866 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-15) [266abdd1] Removing host(s) [] from hosts unseen domain report cache >2014-07-30 16:25:35,868 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-50) [375fcc6] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue >2014-07-30 16:25:35,867 ERROR [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-41) [72233a31] Command org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand throw Vdc Bll exception. With error message VdcBLLException: Cannot allocate IRS server (Failed with error IRS_REPOSITORY_NOT_FOUND and code 5009) >2014-07-30 16:25:35,855 ERROR [org.ovirt.engine.core.bll.storage.GetVmsFromExportDomainQuery] (ajp--127.0.0.1-8702-6) Query GetVmsFromExportDomainQuery failed. Exception message is VdcBLLException: Cannot allocate IRS server (Failed with error IRS_REPOSITORY_NOT_FOUND and code 5009) : org.ovirt.engine.core.common.errors.VdcBLLException: VdcBLLException: Cannot allocate IRS server (Failed with error IRS_REPOSITORY_NOT_FOUND and code 5009): org.ovirt.engine.core.common.errors.VdcBLLException: VdcBLLException: Cannot allocate IRS server (Failed with error IRS_REPOSITORY_NOT_FOUND and code 5009) > at org.ovirt.engine.core.bll.VdsHandler.handleVdsResult(VdsHandler.java:116) [bll.jar:] > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) [bll.jar:] > at org.ovirt.engine.core.bll.storage.GetAllFromExportDomainQuery.executeVerb(GetAllFromExportDomainQuery.java:60) [bll.jar:] > at org.ovirt.engine.core.bll.storage.GetAllFromExportDomainQuery.getAllFromStorage(GetAllFromExportDomainQuery.java:48) [bll.jar:] > at org.ovirt.engine.core.bll.storage.GetAllFromExportDomainQuery.executeQueryCommand(GetAllFromExportDomainQuery.java:37) [bll.jar:] > at org.ovirt.engine.core.bll.QueriesCommandBase.executeCommand(QueriesCommandBase.java:73) [bll.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.bll.Backend.runQueryImpl(Backend.java:492) [bll.jar:] > at org.ovirt.engine.core.bll.Backend.runQuery(Backend.java:466) [bll.jar:] > at sun.reflect.GeneratedMethodAccessor29.invoke(Unknown Source) [:1.7.0_55] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55] > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) [jboss-as-ee-7.1.1.Final.jar:7.1.1.Final] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final] > at org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:374) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final] > at org.jboss.as.weld.ejb.Jsr299BindingsInterceptor.delegateInterception(Jsr299BindingsInterceptor.java:114) [jboss-as-weld-7.1.1.Final.jar:7.1.1.Final] > at org.jboss.as.weld.ejb.Jsr299BindingsInterceptor.doMethodInterception(Jsr299BindingsInterceptor.java:125) [jboss-as-weld-7.1.1.Final.jar:7.1.1.Final] > at org.jboss.as.weld.ejb.Jsr299BindingsInterceptor.processInvocation(Jsr299BindingsInterceptor.java:135) [jboss-as-weld-7.1.1.Final.jar:7.1.1.Final] > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:36) [jboss-as-ee-7.1.1.Final.jar:7.1.1.Final] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final] > at org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:374) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final] > at org.ovirt.engine.core.bll.interceptors.ThreadLocalSessionCleanerInterceptor.injectWebContextToThreadLocal(ThreadLocalSessionCleanerInterceptor.java:13) [bll.jar:] > at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source) [:1.7.0_55] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55] > at org.jboss.as.ee.component.ManagedReferenceLifecycleMethodInterceptorFactory$ManagedReferenceLifecycleMethodInterceptor.processInvocation(ManagedReferenceLifecycleMethodInterceptorFactory.java:123) [jboss-as-ee-7.1.1.Final.jar:7.1.1.Final] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final] > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final] > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:36) [jboss-as-ee-7.1.1.Final.jar:7.1.1.Final] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final] > at org.jboss.as.weld.ejb.EjbRequestScopeActivationInterceptor.processInvocation(EjbRequestScopeActivationInterceptor.java:82) [jboss-as-weld-7.1.1.Final.jar:7.1.1.Final] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final] > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final] > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final] > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) [jboss-as-ee-7.1.1.Final.jar:7.1.1.Final] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final] > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:53) [jboss-as-ejb3-7.1.1.Final.jar:7.1.1.Final] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final] > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:211) [jboss-as-ejb3-7.1.1.Final.jar:7.1.1.Final] > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:363) [jboss-as-ejb3-7.1.1.Final.jar:7.1.1.Final] > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:194) [jboss-as-ejb3-7.1.1.Final.jar:7.1.1.Final] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final] > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) [jboss-as-ejb3-7.1.1.Final.jar:7.1.1.Final] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final] > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) [jboss-as-ejb3-7.1.1.Final.jar:7.1.1.Final] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final] > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) [jboss-as-ee-7.1.1.Final.jar:7.1.1.Final] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final] > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) [jboss-as-ee-7.1.1.Final.jar:7.1.1.Final] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final] > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final] > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) [jboss-as-ee-7.1.1.Final.jar:7.1.1.Final] > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:173) [jboss-as-ee-7.1.1.Final.jar:7.1.1.Final] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final] > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final] > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) [jboss-as-ee-7.1.1.Final.jar:7.1.1.Final] > at org.ovirt.engine.core.common.interfaces.BackendLocal$$$view7.runQuery(Unknown Source) [common.jar:] > at org.ovirt.engine.ui.frontend.server.gwt.GenericApiGWTServiceImpl.runQuery(GenericApiGWTServiceImpl.java:56) > at sun.reflect.GeneratedMethodAccessor64.invoke(Unknown Source) [:1.7.0_55] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55] > at com.google.gwt.rpc.server.RPC.invokeAndStreamResponse(RPC.java:196) > at com.google.gwt.rpc.server.RpcServlet.processCall(RpcServlet.java:172) > at com.google.gwt.rpc.server.RpcServlet.processPost(RpcServlet.java:233) > at com.google.gwt.user.server.rpc.AbstractRemoteServiceServlet.doPost(AbstractRemoteServiceServlet.java:62) > at javax.servlet.http.HttpServlet.service(HttpServlet.java:754) [jboss-servlet-api_3.0_spec-1.0.0.Final.jar:1.0.0.Final] > at javax.servlet.http.HttpServlet.service(HttpServlet.java:847) [jboss-servlet-api_3.0_spec-1.0.0.Final.jar:1.0.0.Final] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:329) > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) > at org.ovirt.engine.core.utils.servlet.HeaderFilter.doFilter(HeaderFilter.java:94) [utils.jar:] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) > at org.ovirt.engine.ui.frontend.server.gwt.GwtCachingFilter.doFilter(GwtCachingFilter.java:132) > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) > at org.ovirt.engine.core.branding.BrandingFilter.doFilter(BrandingFilter.java:72) > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) > at org.ovirt.engine.core.utils.servlet.LocaleFilter.doFilter(LocaleFilter.java:64) [utils.jar:] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) > at org.ovirt.engine.core.aaa.filters.SessionMgmtFilter.doFilter(SessionMgmtFilter.java:31) [aaa.jar:] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) > at org.ovirt.engine.core.aaa.filters.LoginFilter.doFilter(LoginFilter.java:73) [aaa.jar:] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) > at org.ovirt.engine.core.aaa.filters.NegotiationFilter.doFilter(NegotiationFilter.java:105) [aaa.jar:] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) > at org.ovirt.engine.core.aaa.filters.BasicAuthenticationFilter.doFilter(BasicAuthenticationFilter.java:75) [aaa.jar:] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) > at org.ovirt.engine.core.aaa.filters.SessionValidationFilter.doFilter(SessionValidationFilter.java:63) [aaa.jar:] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) > at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:275) > at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:161) > at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:489) > at org.jboss.as.web.security.SecurityContextAssociationValve.invoke(SecurityContextAssociationValve.java:153) > at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:155) > at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102) > at org.jboss.web.rewrite.RewriteValve.invoke(RewriteValve.java:466) > at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109) > at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:368) > at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:505) > at org.apache.coyote.ajp.AjpProtocol$AjpConnectionHandler.process(AjpProtocol.java:445) > at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:930) > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_55] > >2014-07-30 16:25:35,867 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-71) Failed to refresh VDS , vds = 17f7fe56-5f32-40ef-b098-611b84484e0b : rack, VDS Network Error, continuing. >java.net.ConnectException: Connection refused >2014-07-30 16:25:35,880 INFO [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-41) [72233a31] Command [id=00b238d2-2e70-4a3c-b47b-1c976d071d2c]: Compensating CHANGED_STATUS_ONLY of org.ovirt.engine.core.common.businessentities.StoragePoolIsoMap; snapshot: EntityStatusSnapshot [id=storagePoolId = 00000002-0002-0002-0002-0000000001d9, storageId = 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752, status=Inactive]. >2014-07-30 16:25:35,877 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-50) [375fcc6] The following operation Event [eventType=RECONSTRUCT, storagePoolId=00000002-0002-0002-0002-0000000001d9, domainId=71f82fcd-4fa6-4f56-b228-f4fa4bd4f752, vdsId=null, description=IrsBrokerCommand.startReconstruct()] was cancelled, because of reconstruct was run before >2014-07-30 16:25:35,945 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-47) starting processDomainRecovery for domain 44559ec0-6bef-491c-bbd6-e67278f37987:export >2014-07-30 16:25:35,950 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-47) Domain 44559ec0-6bef-491c-bbd6-e67278f37987:export was reported by all hosts in status UP as problematic. Moving the domain to NonOperational. >2014-07-30 16:25:35,955 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-41) [72233a31] Correlation ID: 72233a31, Job ID: 79281f4f-e62b-42ad-8e65-a5ec7b89f594, Call Stack: null, Custom Event ID: -1, Message: Failed to activate Storage Domain 35 (Data Center Default) by admin >2014-07-30 16:25:35,973 INFO [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-47) [b2cf278] Lock Acquired to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE >, sharedLocks= ] >2014-07-30 16:25:35,981 INFO [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-47) [b2cf278] Running command: DeactivateStorageDomainCommand internal: true. Entities affected : ID: 44559ec0-6bef-491c-bbd6-e67278f37987 Type: Storage >2014-07-30 16:25:35,988 INFO [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-47) [b2cf278] Lock freed to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE >, sharedLocks= ] >2014-07-30 16:25:36,012 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-47) [b2cf278] Correlation ID: b2cf278, Job ID: fa4aadbe-e5a7-4068-99ae-8a43e205f9ec, Call Stack: null, Custom Event ID: -1, Message: Storage Domain export (Data Center Default) was deactivated by system because it's not visible by any of the hosts. >2014-07-30 16:25:36,018 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-47) [b2cf278] Removing vds [] from the domain in maintenance cache >2014-07-30 16:25:36,018 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-47) [b2cf278] Removing host(s) [] from hosts unseen domain report cache >2014-07-30 16:25:39,206 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-97) START, GetHardwareInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]), log id: a95296c >2014-07-30 16:25:39,255 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-97) FINISH, GetHardwareInfoVDSCommand, log id: a95296c >2014-07-30 16:25:39,281 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-97) [2f820231] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS >2014-07-30 16:25:39,294 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-97) [57113c30] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-07-30 16:25:39,303 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-97) [5971df3b] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-07-30 16:25:39,316 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-97) [5971df3b] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, storageType = NFS, connectionList = [{ id: 05047c9a-152e-4ef2-b118-4a1753c1eae0, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35export, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 5ee2805a >2014-07-30 16:25:39,378 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-97) [5971df3b] FINISH, ConnectStorageServerVDSCommand, return: {5083f370-af09-457b-9b73-3440e8b65a07=0, 05047c9a-152e-4ef2-b118-4a1753c1eae0=0}, log id: 5ee2805a >2014-07-30 16:25:39,379 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-97) [5971df3b] Host rack storage connection was succeeded >2014-07-30 16:25:39,387 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-7) START, ConnectStoragePoolVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, masterVersion = 1), log id: 2b99de1 >2014-07-30 16:25:45,857 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-72) Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Inactive to Unknown, reason : no reporting hosts >2014-07-30 16:25:45,861 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-72) Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Inactive to Unknown, reason : no reporting hosts >2014-07-30 16:26:39,782 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-7) FINISH, ConnectStoragePoolVDSCommand, log id: 2b99de1 >2014-07-30 16:26:39,796 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-97) [5971df3b] START, SetMOMPolicyParametersVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 4e2d6a98 >2014-07-30 16:26:39,889 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-97) [5971df3b] FINISH, SetMOMPolicyParametersVDSCommand, log id: 4e2d6a98 >2014-07-30 16:26:39,897 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-97) [5971df3b] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host rack was set to Up. >2014-07-30 16:26:39,903 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-97) [5971df3b] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host rack. >2014-07-30 16:26:39,925 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-97) [1b0a6877] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS >2014-07-30 16:26:39,934 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-4) Host 17f7fe56-5f32-40ef-b098-611b84484e0b no longer storage access problem to any relevant domain clearing its report (report id: e8a103cd-8125-46cd-bd80-a65117c56e2f) >2014-07-30 16:26:45,828 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-31) No hosts has reported storage access problem to domains, clearing the handled hosts reports map >2014-07-30 16:26:46,013 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-25) [8cb28d1] hostFromVds::selectedVds - rack, spmStatus Free, storage pool Default >2014-07-30 16:26:46,018 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-25) [8cb28d1] starting spm on vds rack, storage pool Default, prevId -1, LVER -1 >2014-07-30 16:26:46,020 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-25) [8cb28d1] START, SpmStartVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, prevId=-1, prevLVER=-1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 78d02718 >2014-07-30 16:26:46,069 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-25) [8cb28d1] spmStart polling started: taskId = bff5113e-5416-493f-b37f-486fb4a4e9ed >2014-07-30 16:26:47,076 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-25) [8cb28d1] spmStart polling ended: taskId = bff5113e-5416-493f-b37f-486fb4a4e9ed task status = finished >2014-07-30 16:26:47,132 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-25) [8cb28d1] spmStart polling ended, spm status: SPM >2014-07-30 16:26:47,135 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-25) [8cb28d1] START, HSMClearTaskVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, taskId=bff5113e-5416-493f-b37f-486fb4a4e9ed), log id: 3cb14a3e >2014-07-30 16:26:47,180 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-25) [8cb28d1] FINISH, HSMClearTaskVDSCommand, log id: 3cb14a3e >2014-07-30 16:26:47,181 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-25) [8cb28d1] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@6a3d4447, log id: 78d02718 >2014-07-30 16:26:47,186 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-25) [8cb28d1] Initialize Irs proxy from vds: 10.34.62.208 >2014-07-30 16:26:47,193 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-25) [8cb28d1] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208). >2014-07-30 16:26:47,207 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-23) [8cb28d1] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 10637abb >2014-07-30 16:26:47,239 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-23) [8cb28d1] -- executeIrsBrokerCommand: Attempting on storage pool 00000002-0002-0002-0002-0000000001d9 >2014-07-30 16:26:47,241 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-23) [8cb28d1] START, HSMGetAllTasksInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 2b7607c7 >2014-07-30 16:26:47,246 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-23) [8cb28d1] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 2b7607c7 >2014-07-30 16:26:47,247 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-23) [8cb28d1] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 10637abb >2014-07-30 16:26:47,248 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (org.ovirt.thread.pool-8-thread-23) [8cb28d1] Discovered no tasks on Storage Pool Default >2014-07-30 16:39:51,917 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp--127.0.0.1-8702-2) Running command: LoginUserCommand internal: false. >2014-07-30 16:39:51,923 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-2) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-07-30 16:39:51,924 ERROR [org.ovirt.engine.core.aaa.filters.RestApiSessionMgmtFilter] (ajp--127.0.0.1-8702-2) Session-TTL header was not passed. Not setting TTL value >2014-07-30 16:39:52,020 ERROR [org.ovirt.engine.core.aaa.filters.RestApiSessionMgmtFilter] (ajp--127.0.0.1-8702-11) Session-TTL header was not passed. Not setting TTL value >2014-07-30 17:14:57,940 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetVmsInfoVDSCommand] (ajp--127.0.0.1-8702-11) START, GetVmsInfoVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, storageDomainId = 44559ec0-6bef-491c-bbd6-e67278f37987, vmIdList = null), log id: 5b9d755d >2014-07-30 17:14:57,952 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetVmsInfoVDSCommand] (ajp--127.0.0.1-8702-11) FINISH, GetVmsInfoVDSCommand, log id: 5b9d755d >2014-07-30 17:15:00,712 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetVmsInfoVDSCommand] (ajp--127.0.0.1-8702-4) START, GetVmsInfoVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, storageDomainId = 44559ec0-6bef-491c-bbd6-e67278f37987, vmIdList = null), log id: 15e5ed43 >2014-07-30 17:15:00,723 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetVmsInfoVDSCommand] (ajp--127.0.0.1-8702-4) FINISH, GetVmsInfoVDSCommand, log id: 15e5ed43 >2014-07-30 17:23:55,160 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-41) [5d8b5376] Principal admin::internal synchronized >2014-07-30 17:24:22,119 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-100) [52a75f18] Attempting to update VMs/Templates Ovf. >2014-07-30 18:23:55,166 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-14) Principal admin::internal synchronized >2014-07-30 18:24:22,151 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-10) Attempting to update VMs/Templates Ovf. >2014-07-30 19:23:55,172 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-53) Principal admin::internal synchronized >2014-07-30 19:24:22,168 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-37) Attempting to update VMs/Templates Ovf. >2014-07-30 19:26:36,490 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-48) domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 in problem. vds: rack >2014-07-30 19:26:36,493 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-48) domain 44559ec0-6bef-491c-bbd6-e67278f37987:export in problem. vds: rack >2014-07-30 19:28:03,166 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-8) [2e41028] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2014-07-30 19:28:03,171 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-8) [782c75f] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-07-30 19:28:03,175 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-8) [782c75f] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Active to Unknown, reason : null >2014-07-30 19:28:03,177 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-8) [782c75f] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Active to Unknown, reason : null >2014-07-30 19:28:03,190 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-8) [782c75f] Correlation ID: 782c75f, Call Stack: null, Custom Event ID: -1, Message: Invalid status on Data Center Default. Setting Data Center status to Non Responsive (On host rack, Error: Cannot access master storage domain). >2014-07-30 19:30:03,220 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-8) [782c75f] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2014-07-30 19:30:03,224 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-8) [782c75f] hostFromVds::selectedVds - rack, spmStatus returned null! >2014-07-30 19:30:03,225 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-8) [782c75f] IrsBroker::Failed::GetStoragePoolInfoVDS >2014-07-30 19:30:03,226 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-8) [782c75f] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2014-07-30 19:30:03,261 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-30) [6a94c2dc] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage >2014-07-30 19:30:03,280 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-30) [6a94c2dc] START, ResetIrsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, ignoreStopFailed = true), log id: 252f9f2c >2014-07-30 19:30:03,284 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-30) [6a94c2dc] START, SpmStopVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9), log id: 68705a37 >2014-07-30 19:30:03,291 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-30) [6a94c2dc] SpmStopVDSCommand::Stopping SPM on vds rack, pool id 00000002-0002-0002-0002-0000000001d9 >2014-07-30 19:32:03,195 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-76) Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: VDSNetworkException: org.apache.commons.httpclient.NoHttpResponseException: The server 10.34.62.208 failed to respond >2014-07-30 19:32:03,198 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-76) hostFromVds::selectedVds - rack, spmStatus returned null! >2014-07-30 19:32:03,199 WARN [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-30) [6a94c2dc] could not stop spm of pool 00000002-0002-0002-0002-0000000001d9 on vds 17f7fe56-5f32-40ef-b098-611b84484e0b - reason: org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcRunTimeException: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException >2014-07-30 19:32:03,202 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-30) [6a94c2dc] FINISH, SpmStopVDSCommand, log id: 68705a37 >2014-07-30 19:32:03,204 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ListVDSCommand] (DefaultQuartzScheduler_Worker-12) Command ListVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-07-30 19:32:03,206 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-12) Host rack is not responding. It will stay in Connecting state for a grace period of $80 seconds and after that an attempt to fence the host will be issued. >2014-07-30 19:32:03,208 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-30) [6a94c2dc] FINISH, ResetIrsVDSCommand, log id: 252f9f2c >2014-07-30 19:32:03,208 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-12) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:406) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.fetchRunningVms(VdsUpdateRunTimeInfo.java:971) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVmStats(VdsUpdateRunTimeInfo.java:920) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:638) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:235) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) [:1.7.0_55] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_55] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_55] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_55] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_55] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_55] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_55] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_55] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy82.list(Unknown Source) > at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source) [:1.7.0_55] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_55] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_55] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_55] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_55] > >2014-07-30 19:32:03,218 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-30) [6a94c2dc] Correlation ID: 6a94c2dc, Job ID: 2573d5b8-e851-427a-a9d4-6f043c93cfce, Call Stack: null, Custom Event ID: -1, Message: Failed to Reconstruct Master Domain for Data Center Default. >2014-07-30 19:32:03,241 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-9) [782c75f] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue >2014-07-30 19:32:03,241 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-12) Failed to refresh VDS , vds = 17f7fe56-5f32-40ef-b098-611b84484e0b : rack, VDS Network Error, continuing. >java.net.ConnectException: Connection refused >2014-07-30 19:32:03,243 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-39) starting processDomainRecovery for domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 >2014-07-30 19:32:03,300 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-39) Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 was reported by all hosts in status UP as problematic. Not moving the domain to NonOperational because it is being reconstructed now. >2014-07-30 19:32:03,320 WARN [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-39) [16ecab23] CanDoAction of action ReconstructMasterDomain failed. Reasons:VAR__ACTION__RECONSTRUCT_MASTER,VAR__TYPE__STORAGE__DOMAIN,ACTION_TYPE_FAILED_NO_VDS_IN_POOL >2014-07-30 19:32:03,327 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-39) [16ecab23] Removing vds [] from the domain in maintenance cache >2014-07-30 19:32:03,328 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-39) [16ecab23] Removing host(s) [] from hosts unseen domain report cache >2014-07-30 19:32:03,329 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-9) [782c75f] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue >2014-07-30 19:32:03,331 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-36) starting processDomainRecovery for domain 44559ec0-6bef-491c-bbd6-e67278f37987:export >2014-07-30 19:32:03,336 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-36) Domain 44559ec0-6bef-491c-bbd6-e67278f37987:export was reported by all hosts in status UP as problematic. Moving the domain to NonOperational. >2014-07-30 19:32:03,362 INFO [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-36) [18825565] Lock Acquired to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE >, sharedLocks= ] >2014-07-30 19:32:03,368 INFO [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-36) [18825565] Running command: DeactivateStorageDomainCommand internal: true. Entities affected : ID: 44559ec0-6bef-491c-bbd6-e67278f37987 Type: Storage >2014-07-30 19:32:03,374 INFO [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-36) [18825565] Lock freed to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE >, sharedLocks= ] >2014-07-30 19:32:03,382 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-36) [18825565] Correlation ID: 18825565, Job ID: f76c3eb9-55ab-48b6-b0ac-414a5345fafe, Call Stack: null, Custom Event ID: -1, Message: Storage Domain export (Data Center Default) was deactivated by system because it's not visible by any of the hosts. >2014-07-30 19:32:03,387 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-36) [18825565] Removing vds [] from the domain in maintenance cache >2014-07-30 19:32:03,388 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-36) [18825565] Removing host(s) [] from hosts unseen domain report cache >2014-07-30 19:32:06,506 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-89) START, GetHardwareInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]), log id: 54d96ce6 >2014-07-30 19:32:06,566 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-89) FINISH, GetHardwareInfoVDSCommand, log id: 54d96ce6 >2014-07-30 19:32:06,608 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-89) [3514063c] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS >2014-07-30 19:32:06,624 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-89) [12346060] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-07-30 19:32:06,638 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-89) [756c1100] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-07-30 19:32:06,658 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-89) [756c1100] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, storageType = NFS, connectionList = [{ id: 05047c9a-152e-4ef2-b118-4a1753c1eae0, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35export, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: f61d7ad >2014-07-30 19:32:06,675 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-89) [756c1100] FINISH, ConnectStorageServerVDSCommand, return: {5083f370-af09-457b-9b73-3440e8b65a07=0, 05047c9a-152e-4ef2-b118-4a1753c1eae0=0}, log id: f61d7ad >2014-07-30 19:32:06,677 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-89) [756c1100] Host rack storage connection was succeeded >2014-07-30 19:32:06,684 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-19) START, ConnectStoragePoolVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, masterVersion = 1), log id: 4e27dc37 >2014-07-30 19:32:13,203 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-73) Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Inactive to Unknown, reason : no reporting hosts >2014-07-30 19:32:13,207 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-73) Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Inactive to Unknown, reason : no reporting hosts >2014-07-30 19:33:07,061 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-19) FINISH, ConnectStoragePoolVDSCommand, log id: 4e27dc37 >2014-07-30 19:33:07,074 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-89) [756c1100] START, SetMOMPolicyParametersVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 184049e8 >2014-07-30 19:33:07,168 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-89) [756c1100] FINISH, SetMOMPolicyParametersVDSCommand, log id: 184049e8 >2014-07-30 19:33:07,174 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-89) [756c1100] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host rack was set to Up. >2014-07-30 19:33:07,180 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-89) [756c1100] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host rack. >2014-07-30 19:33:07,202 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-89) [1b83a377] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS >2014-07-30 19:33:13,318 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-68) [1fbe99e6] hostFromVds::selectedVds - rack, spmStatus Free, storage pool Default >2014-07-30 19:33:13,323 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-68) [1fbe99e6] starting spm on vds rack, storage pool Default, prevId -1, LVER -1 >2014-07-30 19:33:13,325 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-68) [1fbe99e6] START, SpmStartVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, prevId=-1, prevLVER=-1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 9ae6754 >2014-07-30 19:33:13,373 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-68) [1fbe99e6] spmStart polling started: taskId = 592047e7-270a-46f3-b11d-357579b2bea4 >2014-07-30 19:33:14,385 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-68) [1fbe99e6] spmStart polling ended: taskId = 592047e7-270a-46f3-b11d-357579b2bea4 task status = finished >2014-07-30 19:33:14,441 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-68) [1fbe99e6] spmStart polling ended, spm status: SPM >2014-07-30 19:33:14,444 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-68) [1fbe99e6] START, HSMClearTaskVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, taskId=592047e7-270a-46f3-b11d-357579b2bea4), log id: 3254ecf3 >2014-07-30 19:33:14,488 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-68) [1fbe99e6] FINISH, HSMClearTaskVDSCommand, log id: 3254ecf3 >2014-07-30 19:33:14,489 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-68) [1fbe99e6] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@69eef01b, log id: 9ae6754 >2014-07-30 19:33:14,493 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-68) [1fbe99e6] Initialize Irs proxy from vds: 10.34.62.208 >2014-07-30 19:33:14,499 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-68) [1fbe99e6] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208). >2014-07-30 19:33:14,512 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-49) [1fbe99e6] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: c61f44a >2014-07-30 19:33:14,548 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-49) [1fbe99e6] -- executeIrsBrokerCommand: Attempting on storage pool 00000002-0002-0002-0002-0000000001d9 >2014-07-30 19:33:14,550 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-49) [1fbe99e6] START, HSMGetAllTasksInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 50ad254f >2014-07-30 19:33:14,558 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-49) [1fbe99e6] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 50ad254f >2014-07-30 19:33:14,560 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-49) [1fbe99e6] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: c61f44a >2014-07-30 19:33:14,561 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (org.ovirt.thread.pool-8-thread-49) [1fbe99e6] Discovered no tasks on Storage Pool Default >2014-07-30 20:23:55,181 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-44) [29bdffa0] Principal admin::internal synchronized >2014-07-30 20:24:22,183 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-92) Attempting to update VMs/Templates Ovf. >2014-07-30 21:23:55,189 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-38) [1b9ef0f] Principal admin::internal synchronized >2014-07-30 21:24:22,213 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-43) Attempting to update VMs/Templates Ovf. >2014-07-30 22:23:55,195 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-44) [29bdffa0] Principal admin::internal synchronized >2014-07-30 22:24:22,248 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-92) Attempting to update VMs/Templates Ovf. >2014-07-30 22:32:54,700 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-43) domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 in problem. vds: rack >2014-07-30 22:32:54,702 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-43) domain 44559ec0-6bef-491c-bbd6-e67278f37987:export in problem. vds: rack >2014-07-30 22:34:29,002 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-59) Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2014-07-30 22:34:29,007 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-59) [1f7257b7] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-07-30 22:34:29,010 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-59) [1f7257b7] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Active to Unknown, reason : null >2014-07-30 22:34:29,012 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-59) [1f7257b7] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Active to Unknown, reason : null >2014-07-30 22:34:29,025 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-59) [1f7257b7] Correlation ID: 1f7257b7, Call Stack: null, Custom Event ID: -1, Message: Invalid status on Data Center Default. Setting Data Center status to Non Responsive (On host rack, Error: Cannot access master storage domain). >2014-07-30 22:36:29,055 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-59) [1f7257b7] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2014-07-30 22:36:29,057 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-59) [1f7257b7] hostFromVds::selectedVds - rack, spmStatus returned null! >2014-07-30 22:36:29,058 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-59) [1f7257b7] IrsBroker::Failed::GetStoragePoolInfoVDS >2014-07-30 22:36:29,059 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-59) [1f7257b7] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2014-07-30 22:36:29,102 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-17) [4eb6f81d] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage >2014-07-30 22:36:29,124 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-17) [4eb6f81d] START, ResetIrsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, ignoreStopFailed = true), log id: 1c8da596 >2014-07-30 22:36:29,128 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-17) [4eb6f81d] START, SpmStopVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9), log id: 6f26c4a6 >2014-07-30 22:36:29,134 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-17) [4eb6f81d] SpmStopVDSCommand::Stopping SPM on vds rack, pool id 00000002-0002-0002-0002-0000000001d9 >2014-07-30 22:38:29,028 WARN [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-17) [4eb6f81d] could not stop spm of pool 00000002-0002-0002-0002-0000000001d9 on vds 17f7fe56-5f32-40ef-b098-611b84484e0b - reason: org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcRunTimeException: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException >2014-07-30 22:38:29,035 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetStatsVDSCommand] (DefaultQuartzScheduler_Worker-65) Command GetStatsVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-07-30 22:38:29,038 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-17) [4eb6f81d] FINISH, SpmStopVDSCommand, log id: 6f26c4a6 >2014-07-30 22:38:29,039 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-26) [6bd9692] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: VDSNetworkException: org.apache.commons.httpclient.NoHttpResponseException: The server 10.34.62.208 failed to respond >2014-07-30 22:38:29,041 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-26) [6bd9692] hostFromVds::selectedVds - rack, spmStatus returned null! >2014-07-30 22:38:29,040 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-65) Host rack is not responding. It will stay in Connecting state for a grace period of $80 seconds and after that an attempt to fence the host will be issued. >2014-07-30 22:38:29,043 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-65) vds::refreshVdsStats Failed getVdsStats, vds = 17f7fe56-5f32-40ef-b098-611b84484e0b : rack, error = VDSNetworkException: java.net.ConnectException: Connection refused >2014-07-30 22:38:29,045 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-65) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsStats(VdsUpdateRunTimeInfo.java:669) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:623) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:235) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) [:1.7.0_55] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_55] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_55] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_55] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_55] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_55] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_55] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_55] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy82.getVdsStats(Unknown Source) > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_55] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_55] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_55] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_55] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_55] > >2014-07-30 22:38:29,051 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-17) [4eb6f81d] FINISH, ResetIrsVDSCommand, log id: 1c8da596 >2014-07-30 22:38:29,091 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-65) Failed to refresh VDS , vds = 17f7fe56-5f32-40ef-b098-611b84484e0b : rack, VDS Network Error, continuing. >java.net.ConnectException: Connection refused >2014-07-30 22:38:29,094 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-17) [4eb6f81d] Correlation ID: 4eb6f81d, Job ID: 2ecec18f-a4b2-4caa-84c8-ac9cbd7dc151, Call Stack: null, Custom Event ID: -1, Message: Failed to Reconstruct Master Domain for Data Center Default. >2014-07-30 22:38:29,104 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-28) [1f7257b7] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue >2014-07-30 22:38:29,107 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-1) starting processDomainRecovery for domain 44559ec0-6bef-491c-bbd6-e67278f37987:export >2014-07-30 22:38:29,178 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-1) Domain 44559ec0-6bef-491c-bbd6-e67278f37987:export was reported by all hosts in status UP as problematic. Moving the domain to NonOperational. >2014-07-30 22:38:29,199 INFO [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-1) [3e307027] Lock Acquired to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE >, sharedLocks= ] >2014-07-30 22:38:29,207 INFO [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-1) [3e307027] Running command: DeactivateStorageDomainCommand internal: true. Entities affected : ID: 44559ec0-6bef-491c-bbd6-e67278f37987 Type: Storage >2014-07-30 22:38:29,213 INFO [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-1) [3e307027] Lock freed to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE >, sharedLocks= ] >2014-07-30 22:38:29,222 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-1) [3e307027] Correlation ID: 3e307027, Job ID: b247f619-daf3-4b7b-893a-9aa235f98fa9, Call Stack: null, Custom Event ID: -1, Message: Storage Domain export (Data Center Default) was deactivated by system because it's not visible by any of the hosts. >2014-07-30 22:38:29,227 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-1) [3e307027] Removing vds [] from the domain in maintenance cache >2014-07-30 22:38:29,228 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-1) [3e307027] Removing host(s) [] from hosts unseen domain report cache >2014-07-30 22:38:29,229 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-12) starting processDomainRecovery for domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 >2014-07-30 22:38:29,234 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-12) Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 was reported by all hosts in status UP as problematic. Not moving the domain to NonOperational because it is being reconstructed now. >2014-07-30 22:38:29,254 WARN [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-12) [47b016c8] CanDoAction of action ReconstructMasterDomain failed. Reasons:VAR__ACTION__RECONSTRUCT_MASTER,VAR__TYPE__STORAGE__DOMAIN,ACTION_TYPE_FAILED_NO_VDS_IN_POOL >2014-07-30 22:38:29,261 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-12) [47b016c8] Removing vds [] from the domain in maintenance cache >2014-07-30 22:38:29,261 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-12) [47b016c8] Removing host(s) [] from hosts unseen domain report cache >2014-07-30 22:38:29,262 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-28) [1f7257b7] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue >2014-07-30 22:38:32,355 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-68) [1fbe99e6] START, GetHardwareInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]), log id: 35aa52cc >2014-07-30 22:38:32,406 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-68) [1fbe99e6] FINISH, GetHardwareInfoVDSCommand, log id: 35aa52cc >2014-07-30 22:38:32,435 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-68) [1aa24b07] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS >2014-07-30 22:38:32,486 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-68) [6a81647c] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-07-30 22:38:32,494 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-68) [9604479] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-07-30 22:38:32,507 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-68) [9604479] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, storageType = NFS, connectionList = [{ id: 05047c9a-152e-4ef2-b118-4a1753c1eae0, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35export, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 64dac9e5 >2014-07-30 22:38:32,571 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-68) [9604479] FINISH, ConnectStorageServerVDSCommand, return: {5083f370-af09-457b-9b73-3440e8b65a07=0, 05047c9a-152e-4ef2-b118-4a1753c1eae0=0}, log id: 64dac9e5 >2014-07-30 22:38:32,573 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-68) [9604479] Host rack storage connection was succeeded >2014-07-30 22:38:32,582 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-48) START, ConnectStoragePoolVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, masterVersion = 1), log id: 4adb6926 >2014-07-30 22:38:39,046 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-2) Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Inactive to Unknown, reason : no reporting hosts >2014-07-30 22:38:39,051 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-2) Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Inactive to Unknown, reason : no reporting hosts >2014-07-30 22:39:32,982 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-48) FINISH, ConnectStoragePoolVDSCommand, log id: 4adb6926 >2014-07-30 22:39:32,995 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-68) [9604479] START, SetMOMPolicyParametersVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 38598c78 >2014-07-30 22:39:33,089 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-68) [9604479] FINISH, SetMOMPolicyParametersVDSCommand, log id: 38598c78 >2014-07-30 22:39:33,098 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-68) [9604479] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host rack was set to Up. >2014-07-30 22:39:33,104 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-68) [9604479] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host rack. >2014-07-30 22:39:33,120 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-68) [6dc0d36f] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS >2014-07-30 22:39:39,216 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-81) hostFromVds::selectedVds - rack, spmStatus Free, storage pool Default >2014-07-30 22:39:39,223 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-81) starting spm on vds rack, storage pool Default, prevId -1, LVER -1 >2014-07-30 22:39:39,225 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-81) START, SpmStartVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, prevId=-1, prevLVER=-1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 3b575257 >2014-07-30 22:39:39,273 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-81) spmStart polling started: taskId = 694f6db5-2ed4-4757-86df-bcd06c31407d >2014-07-30 22:39:40,280 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-81) spmStart polling ended: taskId = 694f6db5-2ed4-4757-86df-bcd06c31407d task status = finished >2014-07-30 22:39:40,335 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-81) spmStart polling ended, spm status: SPM >2014-07-30 22:39:40,338 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-81) START, HSMClearTaskVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, taskId=694f6db5-2ed4-4757-86df-bcd06c31407d), log id: 41465994 >2014-07-30 22:39:40,382 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-81) FINISH, HSMClearTaskVDSCommand, log id: 41465994 >2014-07-30 22:39:40,383 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-81) FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@7d51d68d, log id: 3b575257 >2014-07-30 22:39:40,387 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-81) Initialize Irs proxy from vds: 10.34.62.208 >2014-07-30 22:39:40,393 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-81) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208). >2014-07-30 22:39:40,403 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-16) START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 5888787f >2014-07-30 22:39:40,437 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-16) -- executeIrsBrokerCommand: Attempting on storage pool 00000002-0002-0002-0002-0000000001d9 >2014-07-30 22:39:40,439 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-16) START, HSMGetAllTasksInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 4b80b251 >2014-07-30 22:39:40,444 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-16) FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 4b80b251 >2014-07-30 22:39:40,445 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-16) FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 5888787f >2014-07-30 22:39:40,446 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (org.ovirt.thread.pool-8-thread-16) Discovered no tasks on Storage Pool Default >2014-07-30 23:23:55,201 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-15) Principal admin::internal synchronized >2014-07-30 23:24:22,262 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-5) Attempting to update VMs/Templates Ovf. >2014-07-31 00:23:55,206 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-26) [6bd9692] Principal admin::internal synchronized >2014-07-31 00:24:22,293 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-20) [21e62018] Attempting to update VMs/Templates Ovf. >2014-07-31 01:23:55,214 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-49) Principal admin::internal synchronized >2014-07-31 01:24:22,311 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-61) [58bfe349] Attempting to update VMs/Templates Ovf. >2014-07-31 01:39:26,958 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-2) domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 in problem. vds: rack >2014-07-31 01:39:26,959 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-2) domain 44559ec0-6bef-491c-bbd6-e67278f37987:export in problem. vds: rack >2014-07-31 01:40:56,861 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-23) Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2014-07-31 01:40:56,865 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-23) [11b7484a] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-07-31 01:40:56,869 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-23) [11b7484a] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Active to Unknown, reason : null >2014-07-31 01:40:56,872 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-23) [11b7484a] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Active to Unknown, reason : null >2014-07-31 01:40:56,888 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-23) [11b7484a] Correlation ID: 11b7484a, Call Stack: null, Custom Event ID: -1, Message: Invalid status on Data Center Default. Setting Data Center status to Non Responsive (On host rack, Error: Cannot access master storage domain). >2014-07-31 01:42:56,928 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-23) [11b7484a] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2014-07-31 01:42:56,930 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-23) [11b7484a] hostFromVds::selectedVds - rack, spmStatus returned null! >2014-07-31 01:42:56,931 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-23) [11b7484a] IrsBroker::Failed::GetStoragePoolInfoVDS >2014-07-31 01:42:56,932 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-23) [11b7484a] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2014-07-31 01:42:56,969 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-30) [251e5084] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage >2014-07-31 01:42:56,989 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-30) [251e5084] START, ResetIrsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, ignoreStopFailed = true), log id: 65247c5e >2014-07-31 01:42:56,992 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-30) [251e5084] START, SpmStopVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9), log id: 36aa66fb >2014-07-31 01:42:56,998 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-30) [251e5084] SpmStopVDSCommand::Stopping SPM on vds rack, pool id 00000002-0002-0002-0002-0000000001d9 >2014-07-31 01:44:56,886 WARN [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-30) [251e5084] could not stop spm of pool 00000002-0002-0002-0002-0000000001d9 on vds 17f7fe56-5f32-40ef-b098-611b84484e0b - reason: org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcRunTimeException: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException >2014-07-31 01:44:56,888 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-30) [251e5084] FINISH, SpmStopVDSCommand, log id: 36aa66fb >2014-07-31 01:44:56,894 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ListVDSCommand] (DefaultQuartzScheduler_Worker-68) [6dc0d36f] Command ListVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-07-31 01:44:56,896 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-68) [6dc0d36f] Host rack is not responding. It will stay in Connecting state for a grace period of $80 seconds and after that an attempt to fence the host will be issued. >2014-07-31 01:44:56,899 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-68) [6dc0d36f] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:406) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.fetchRunningVms(VdsUpdateRunTimeInfo.java:971) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVmStats(VdsUpdateRunTimeInfo.java:920) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:638) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:235) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) [:1.7.0_55] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_55] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_55] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_55] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_55] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_55] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_55] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_55] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy82.list(Unknown Source) > at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source) [:1.7.0_55] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_55] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_55] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_55] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_55] > >2014-07-31 01:44:56,899 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-30) [251e5084] FINISH, ResetIrsVDSCommand, log id: 65247c5e >2014-07-31 01:44:56,897 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-15) Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: VDSNetworkException: org.apache.commons.httpclient.NoHttpResponseException: The server 10.34.62.208 failed to respond >2014-07-31 01:44:56,931 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-15) hostFromVds::selectedVds - rack, spmStatus returned null! >2014-07-31 01:44:56,939 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-68) [6dc0d36f] Failed to refresh VDS , vds = 17f7fe56-5f32-40ef-b098-611b84484e0b : rack, VDS Network Error, continuing. >java.net.ConnectException: Connection refused >2014-07-31 01:44:56,943 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-30) [251e5084] Correlation ID: 251e5084, Job ID: 181abcb7-3270-4342-b5c3-bdd8db678191, Call Stack: null, Custom Event ID: -1, Message: Failed to Reconstruct Master Domain for Data Center Default. >2014-07-31 01:44:56,948 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-18) [11b7484a] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue >2014-07-31 01:44:56,950 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-38) starting processDomainRecovery for domain 44559ec0-6bef-491c-bbd6-e67278f37987:export >2014-07-31 01:44:57,010 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-38) Domain 44559ec0-6bef-491c-bbd6-e67278f37987:export was reported by all hosts in status UP as problematic. Moving the domain to NonOperational. >2014-07-31 01:44:57,027 INFO [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-38) [3b951098] Lock Acquired to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE >, sharedLocks= ] >2014-07-31 01:44:57,034 INFO [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-38) [3b951098] Running command: DeactivateStorageDomainCommand internal: true. Entities affected : ID: 44559ec0-6bef-491c-bbd6-e67278f37987 Type: Storage >2014-07-31 01:44:57,039 INFO [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-38) [3b951098] Lock freed to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE >, sharedLocks= ] >2014-07-31 01:44:57,048 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-38) [3b951098] Correlation ID: 3b951098, Job ID: b5b3da83-baef-485a-be12-654af7baba83, Call Stack: null, Custom Event ID: -1, Message: Storage Domain export (Data Center Default) was deactivated by system because it's not visible by any of the hosts. >2014-07-31 01:44:57,053 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-38) [3b951098] Removing vds [] from the domain in maintenance cache >2014-07-31 01:44:57,053 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-38) [3b951098] Removing host(s) [] from hosts unseen domain report cache >2014-07-31 01:44:57,055 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-24) starting processDomainRecovery for domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 >2014-07-31 01:44:57,060 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-24) Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 was reported by all hosts in status UP as problematic. Not moving the domain to NonOperational because it is being reconstructed now. >2014-07-31 01:44:57,078 WARN [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-24) [11e6cdd1] CanDoAction of action ReconstructMasterDomain failed. Reasons:VAR__ACTION__RECONSTRUCT_MASTER,VAR__TYPE__STORAGE__DOMAIN,ACTION_TYPE_FAILED_NO_VDS_IN_POOL >2014-07-31 01:44:57,084 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-24) [11e6cdd1] Removing vds [] from the domain in maintenance cache >2014-07-31 01:44:57,085 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-24) [11e6cdd1] Removing host(s) [] from hosts unseen domain report cache >2014-07-31 01:44:57,086 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-18) [11b7484a] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue >2014-07-31 01:45:00,014 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-77) Autorecovering 2 storage domains >2014-07-31 01:45:00,015 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-77) Autorecovering storage domains id: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 >2014-07-31 01:45:00,017 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-77) [68befb52] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage >2014-07-31 01:45:00,018 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-77) [68befb52] ConnectDomainToStorage. Before Connect all hosts to pool. Time:7/31/14 1:45 AM >2014-07-31 01:45:00,026 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-77) [68befb52] ConnectDomainToStorage. After Connect all hosts to pool. Time:7/31/14 1:45 AM >2014-07-31 01:45:00,029 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-77) [68befb52] Autorecovering storage domains id: 44559ec0-6bef-491c-bbd6-e67278f37987 >2014-07-31 01:45:00,034 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-77) [4df186e8] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 44559ec0-6bef-491c-bbd6-e67278f37987 Type: Storage >2014-07-31 01:45:00,035 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-77) [4df186e8] ConnectDomainToStorage. Before Connect all hosts to pool. Time:7/31/14 1:45 AM >2014-07-31 01:45:00,042 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-77) [4df186e8] ConnectDomainToStorage. After Connect all hosts to pool. Time:7/31/14 1:45 AM >2014-07-31 01:45:00,045 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-52) Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@6e9ec75f >2014-07-31 01:45:00,047 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-52) HostName = rack >2014-07-31 01:45:00,049 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-52) Command GetCapabilitiesVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]) execution failed. Exception: VDSRecoveringException: Recovering from crash or Initializing >2014-07-31 01:45:00,064 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-52) Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSRecoveringException: Recovering from crash or Initializing > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:16) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:566) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:235) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Host rack is initializing. Message: Recovering from crash or Initializing >2014-07-31 01:45:00,071 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-52) Failed to refresh VDS , vds = 17f7fe56-5f32-40ef-b098-611b84484e0b : rack, error = Recovering from crash or Initializing, continuing. >2014-07-31 01:45:03,318 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-20) [21e62018] START, GetHardwareInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]), log id: 69d71ab6 >2014-07-31 01:45:03,372 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-20) [21e62018] FINISH, GetHardwareInfoVDSCommand, log id: 69d71ab6 >2014-07-31 01:45:03,399 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-20) [2dbcdcde] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS >2014-07-31 01:45:03,411 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-20) [2dbcdcde] Cancelling the recovery from crash timer for VDS 17f7fe56-5f32-40ef-b098-611b84484e0b because vds started initializing >2014-07-31 01:45:03,418 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-20) [4267262c] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-07-31 01:45:03,433 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-20) [5e41ac3d] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-07-31 01:45:03,444 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-20) [5e41ac3d] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, storageType = NFS, connectionList = [{ id: 05047c9a-152e-4ef2-b118-4a1753c1eae0, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35export, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 4f3531e0 >2014-07-31 01:45:03,467 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-20) [5e41ac3d] FINISH, ConnectStorageServerVDSCommand, return: {5083f370-af09-457b-9b73-3440e8b65a07=0, 05047c9a-152e-4ef2-b118-4a1753c1eae0=0}, log id: 4f3531e0 >2014-07-31 01:45:03,468 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-20) [5e41ac3d] Host rack storage connection was succeeded >2014-07-31 01:45:03,476 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-35) START, ConnectStoragePoolVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, masterVersion = 1), log id: 1fcccaa2 >2014-07-31 01:45:06,936 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-33) Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Inactive to Unknown, reason : no reporting hosts >2014-07-31 01:45:06,940 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-33) Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Inactive to Unknown, reason : no reporting hosts >2014-07-31 01:46:03,880 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-35) FINISH, ConnectStoragePoolVDSCommand, log id: 1fcccaa2 >2014-07-31 01:46:03,894 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-20) [5e41ac3d] START, SetMOMPolicyParametersVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 7f2e4816 >2014-07-31 01:46:03,988 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-20) [5e41ac3d] FINISH, SetMOMPolicyParametersVDSCommand, log id: 7f2e4816 >2014-07-31 01:46:03,998 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-20) [5e41ac3d] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host rack was set to Up. >2014-07-31 01:46:04,005 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-20) [5e41ac3d] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host rack. >2014-07-31 01:46:04,037 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-20) [3adf532b] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS >2014-07-31 01:46:07,113 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-69) [52d379dc] hostFromVds::selectedVds - rack, spmStatus Free, storage pool Default >2014-07-31 01:46:07,119 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-69) [52d379dc] starting spm on vds rack, storage pool Default, prevId -1, LVER -1 >2014-07-31 01:46:07,121 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-69) [52d379dc] START, SpmStartVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, prevId=-1, prevLVER=-1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 32d2768 >2014-07-31 01:46:07,169 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-69) [52d379dc] spmStart polling started: taskId = 19ff676c-b136-4ec9-8767-8b277c26fcaf >2014-07-31 01:46:08,176 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-69) [52d379dc] spmStart polling ended: taskId = 19ff676c-b136-4ec9-8767-8b277c26fcaf task status = finished >2014-07-31 01:46:08,233 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-69) [52d379dc] spmStart polling ended, spm status: SPM >2014-07-31 01:46:08,235 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-69) [52d379dc] START, HSMClearTaskVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, taskId=19ff676c-b136-4ec9-8767-8b277c26fcaf), log id: 3bf7656a >2014-07-31 01:46:08,282 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-69) [52d379dc] FINISH, HSMClearTaskVDSCommand, log id: 3bf7656a >2014-07-31 01:46:08,283 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-69) [52d379dc] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@7e5cfe61, log id: 32d2768 >2014-07-31 01:46:08,287 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-69) [52d379dc] Initialize Irs proxy from vds: 10.34.62.208 >2014-07-31 01:46:08,293 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-69) [52d379dc] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208). >2014-07-31 01:46:08,295 WARN [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-8-thread-27) Executing a command: java.util.concurrent.FutureTask , but note that there are 1 tasks in the queue. >2014-07-31 01:46:08,323 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-27) [52d379dc] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 7fa108de >2014-07-31 01:46:08,344 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-27) [52d379dc] -- executeIrsBrokerCommand: Attempting on storage pool 00000002-0002-0002-0002-0000000001d9 >2014-07-31 01:46:08,347 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-27) [52d379dc] START, HSMGetAllTasksInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 300a2729 >2014-07-31 01:46:08,354 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-27) [52d379dc] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 300a2729 >2014-07-31 01:46:08,356 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-27) [52d379dc] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 7fa108de >2014-07-31 01:46:08,358 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (org.ovirt.thread.pool-8-thread-27) [52d379dc] Discovered no tasks on Storage Pool Default >2014-07-31 02:23:55,225 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-38) [1b9ef0f] Principal admin::internal synchronized >2014-07-31 02:24:22,344 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-83) Attempting to update VMs/Templates Ovf. >2014-07-31 03:23:55,234 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-12) Principal admin::internal synchronized >2014-07-31 03:24:22,358 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-66) [2734597c] Attempting to update VMs/Templates Ovf. >2014-07-31 03:35:35,000 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-46) [56f1581c] Start deleteAgedOutCommandEntities >2014-07-31 03:35:35,001 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-35) Start deleteAgedOutAuditLogs >2014-07-31 03:35:35,006 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-46) [56f1581c] Finished deleteAgedOutCommandEntities >2014-07-31 03:35:35,010 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-35) Finished deleteAgedOutAuditLogs >2014-07-31 04:23:55,241 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-5) Principal admin::internal synchronized >2014-07-31 04:24:22,392 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-21) Attempting to update VMs/Templates Ovf. >2014-07-31 04:45:45,100 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-5) domain 44559ec0-6bef-491c-bbd6-e67278f37987:export in problem. vds: rack >2014-07-31 04:46:00,213 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-41) domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 in problem. vds: rack >2014-07-31 04:47:26,860 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-13) [4dca0641] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2014-07-31 04:47:26,865 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-13) [519c108f] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-07-31 04:47:26,869 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-13) [519c108f] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Active to Unknown, reason : null >2014-07-31 04:47:26,871 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-13) [519c108f] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Active to Unknown, reason : null >2014-07-31 04:47:26,888 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-13) [519c108f] Correlation ID: 519c108f, Call Stack: null, Custom Event ID: -1, Message: Invalid status on Data Center Default. Setting Data Center status to Non Responsive (On host rack, Error: Cannot access master storage domain). >2014-07-31 04:49:26,919 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-13) [519c108f] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2014-07-31 04:49:26,922 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-13) [519c108f] hostFromVds::selectedVds - rack, spmStatus returned null! >2014-07-31 04:49:26,923 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-13) [519c108f] IrsBroker::Failed::GetStoragePoolInfoVDS >2014-07-31 04:49:26,924 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-13) [519c108f] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2014-07-31 04:49:26,969 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-31) [320efffb] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage >2014-07-31 04:49:26,985 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-31) [320efffb] START, ResetIrsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, ignoreStopFailed = true), log id: 1f761dbc >2014-07-31 04:49:26,989 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-31) [320efffb] START, SpmStopVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9), log id: 5f1fb654 >2014-07-31 04:49:27,000 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-31) [320efffb] SpmStopVDSCommand::Stopping SPM on vds rack, pool id 00000002-0002-0002-0002-0000000001d9 >2014-07-31 04:50:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-45) Autorecovering 1 storage domains >2014-07-31 04:50:00,005 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-45) Autorecovering storage domains id: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 >2014-07-31 04:50:00,007 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-45) [51d0e706] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage >2014-07-31 04:50:00,009 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-45) [51d0e706] ConnectDomainToStorage. Before Connect all hosts to pool. Time:7/31/14 4:50 AM >2014-07-31 04:50:00,023 INFO [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-8-thread-26) [1ba7403c] Running command: ConnectStorageToVdsCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2014-07-31 04:50:00,027 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-26) [1ba7403c] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 64111a43 >2014-07-31 04:51:26,908 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-58) [7b19edd8] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2014-07-31 04:51:26,911 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-58) [7b19edd8] hostFromVds::selectedVds - rack, spmStatus returned null! >2014-07-31 04:51:26,912 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-58) [7b19edd8] IrsBroker::Failed::GetStoragePoolInfoVDS >2014-07-31 04:51:26,913 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-58) [7b19edd8] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2014-07-31 04:52:27,002 WARN [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-31) [320efffb] could not stop spm of pool 00000002-0002-0002-0002-0000000001d9 on vds 17f7fe56-5f32-40ef-b098-611b84484e0b - reason: org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcRunTimeException: java.util.concurrent.TimeoutException >2014-07-31 04:52:27,007 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-31) [320efffb] FINISH, SpmStopVDSCommand, log id: 5f1fb654 >2014-07-31 04:52:27,013 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-31) [320efffb] FINISH, ResetIrsVDSCommand, log id: 1f761dbc >2014-07-31 04:52:27,024 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-26) [1ba7403c] FINISH, ConnectStorageServerVDSCommand, return: {5083f370-af09-457b-9b73-3440e8b65a07=0}, log id: 64111a43 >2014-07-31 04:52:27,027 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-45) [51d0e706] ConnectDomainToStorage. After Connect all hosts to pool. Time:7/31/14 4:52 AM >2014-07-31 04:52:27,030 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-31) [320efffb] Correlation ID: 320efffb, Job ID: 31346ad7-d12d-4841-b086-b78a4c76f327, Call Stack: null, Custom Event ID: -1, Message: Failed to Reconstruct Master Domain for Data Center Default. >2014-07-31 04:52:27,038 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-25) [519c108f] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue >2014-07-31 04:52:27,040 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-18) starting processDomainRecovery for domain 44559ec0-6bef-491c-bbd6-e67278f37987:export >2014-07-31 04:52:27,106 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-18) Domain 44559ec0-6bef-491c-bbd6-e67278f37987:export was reported by all hosts in status UP as problematic. Moving the domain to NonOperational. >2014-07-31 04:52:27,129 INFO [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-18) [9f67958] Lock Acquired to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE >, sharedLocks= ] >2014-07-31 04:52:27,136 INFO [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-18) [9f67958] Running command: DeactivateStorageDomainCommand internal: true. Entities affected : ID: 44559ec0-6bef-491c-bbd6-e67278f37987 Type: Storage >2014-07-31 04:52:27,144 INFO [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-18) [9f67958] Lock freed to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE >, sharedLocks= ] >2014-07-31 04:52:27,157 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-18) [9f67958] Correlation ID: 9f67958, Job ID: d85583c2-b16d-4e16-aa80-f0f342579705, Call Stack: null, Custom Event ID: -1, Message: Storage Domain export (Data Center Default) was deactivated by system because it's not visible by any of the hosts. >2014-07-31 04:52:27,164 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-18) [9f67958] Removing vds [] from the domain in maintenance cache >2014-07-31 04:52:27,166 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-18) [9f67958] Removing host(s) [] from hosts unseen domain report cache >2014-07-31 04:52:27,169 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-42) starting processDomainRecovery for domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 >2014-07-31 04:52:27,177 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-42) Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 was reported by all hosts in status UP as problematic. Not moving the domain to NonOperational because it is being reconstructed now. >2014-07-31 04:52:27,208 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-42) [4aca6127] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage >2014-07-31 04:52:27,233 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-42) [4aca6127] Removing vds [] from the domain in maintenance cache >2014-07-31 04:52:27,234 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-42) [4aca6127] Removing host(s) [] from hosts unseen domain report cache >2014-07-31 04:52:27,235 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-25) [519c108f] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue >2014-07-31 04:53:26,931 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-2) Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: VDSNetworkException: org.apache.commons.httpclient.NoHttpResponseException: The server 10.34.62.208 failed to respond >2014-07-31 04:53:26,933 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-2) hostFromVds::selectedVds - rack, spmStatus returned null! >2014-07-31 04:53:27,687 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ListVDSCommand] (DefaultQuartzScheduler_Worker-15) Command ListVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]) execution failed. Exception: VDSRecoveringException: Recovering from crash or Initializing >2014-07-31 04:53:27,689 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-15) Failed vds listing, vds = 17f7fe56-5f32-40ef-b098-611b84484e0b : rack, error = org.ovirt.engine.core.vdsbroker.vdsbroker.VDSRecoveringException: Recovering from crash or Initializing >2014-07-31 04:53:27,707 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-15) Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSRecoveringException: Recovering from crash or Initializing > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.vdsbroker.ListVDSCommand.executeVdsBrokerCommand(ListVDSCommand.java:24) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:406) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.fetchRunningVms(VdsUpdateRunTimeInfo.java:971) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVmStats(VdsUpdateRunTimeInfo.java:920) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:638) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:235) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Host rack is initializing. Message: Recovering from crash or Initializing >2014-07-31 04:53:27,715 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-15) Failed to refresh VDS , vds = 17f7fe56-5f32-40ef-b098-611b84484e0b : rack, error = Recovering from crash or Initializing, continuing. >2014-07-31 04:53:30,950 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-77) [4df186e8] START, GetHardwareInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]), log id: 5bfde9d9 >2014-07-31 04:53:30,998 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-77) [4df186e8] FINISH, GetHardwareInfoVDSCommand, log id: 5bfde9d9 >2014-07-31 04:53:31,025 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-77) [53909aaa] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS >2014-07-31 04:53:31,074 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-77) [53909aaa] Cancelling the recovery from crash timer for VDS 17f7fe56-5f32-40ef-b098-611b84484e0b because vds started initializing >2014-07-31 04:53:31,076 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-77) [8936159] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-07-31 04:53:31,083 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-77) [4be6c79c] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-07-31 04:53:31,095 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-77) [4be6c79c] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, storageType = NFS, connectionList = [{ id: 05047c9a-152e-4ef2-b118-4a1753c1eae0, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35export, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 6153fd97 >2014-07-31 04:53:31,160 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-77) [4be6c79c] FINISH, ConnectStorageServerVDSCommand, return: {5083f370-af09-457b-9b73-3440e8b65a07=0, 05047c9a-152e-4ef2-b118-4a1753c1eae0=0}, log id: 6153fd97 >2014-07-31 04:53:31,162 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-77) [4be6c79c] Host rack storage connection was succeeded >2014-07-31 04:53:31,171 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-2) START, ConnectStoragePoolVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, masterVersion = 1), log id: 543fc72a >2014-07-31 04:53:36,937 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-50) [375fcc6] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Inactive to Unknown, reason : no reporting hosts >2014-07-31 04:53:36,941 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-50) [375fcc6] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Inactive to Unknown, reason : no reporting hosts >2014-07-31 04:54:31,619 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-2) FINISH, ConnectStoragePoolVDSCommand, log id: 543fc72a >2014-07-31 04:54:31,632 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-77) [4be6c79c] START, SetMOMPolicyParametersVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 1841609e >2014-07-31 04:54:31,727 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-77) [4be6c79c] FINISH, SetMOMPolicyParametersVDSCommand, log id: 1841609e >2014-07-31 04:54:31,741 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-77) [4be6c79c] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host rack was set to Up. >2014-07-31 04:54:31,750 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-77) [4be6c79c] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host rack. >2014-07-31 04:54:31,770 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-77) [79b9f7da] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS >2014-07-31 04:54:37,059 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-7) [2b0f39f8] hostFromVds::selectedVds - rack, spmStatus Free, storage pool Default >2014-07-31 04:54:37,064 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-7) [2b0f39f8] starting spm on vds rack, storage pool Default, prevId -1, LVER -1 >2014-07-31 04:54:37,066 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-7) [2b0f39f8] START, SpmStartVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, prevId=-1, prevLVER=-1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 714e82ad >2014-07-31 04:54:37,116 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-7) [2b0f39f8] spmStart polling started: taskId = 5fa4711d-b918-4774-b314-552fbb728ca5 >2014-07-31 04:54:38,122 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-7) [2b0f39f8] spmStart polling ended: taskId = 5fa4711d-b918-4774-b314-552fbb728ca5 task status = finished >2014-07-31 04:54:38,178 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-7) [2b0f39f8] spmStart polling ended, spm status: SPM >2014-07-31 04:54:38,180 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-7) [2b0f39f8] START, HSMClearTaskVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, taskId=5fa4711d-b918-4774-b314-552fbb728ca5), log id: 3ddcf217 >2014-07-31 04:54:38,225 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-7) [2b0f39f8] FINISH, HSMClearTaskVDSCommand, log id: 3ddcf217 >2014-07-31 04:54:38,226 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-7) [2b0f39f8] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@7e3b69c7, log id: 714e82ad >2014-07-31 04:54:38,230 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-7) [2b0f39f8] Initialize Irs proxy from vds: 10.34.62.208 >2014-07-31 04:54:38,236 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-7) [2b0f39f8] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208). >2014-07-31 04:54:38,249 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-13) [2b0f39f8] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 56049f5c >2014-07-31 04:54:38,288 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-13) [2b0f39f8] -- executeIrsBrokerCommand: Attempting on storage pool 00000002-0002-0002-0002-0000000001d9 >2014-07-31 04:54:38,290 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-13) [2b0f39f8] START, HSMGetAllTasksInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 7a516f5 >2014-07-31 04:54:38,296 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-13) [2b0f39f8] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 7a516f5 >2014-07-31 04:54:38,298 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-13) [2b0f39f8] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 56049f5c >2014-07-31 04:54:38,299 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (org.ovirt.thread.pool-8-thread-13) [2b0f39f8] Discovered no tasks on Storage Pool Default >2014-07-31 05:23:55,247 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-36) [3e2ffa8] Principal admin::internal synchronized >2014-07-31 05:24:22,409 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-66) [2734597c] Attempting to update VMs/Templates Ovf. >2014-07-31 06:23:55,254 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-47) Principal admin::internal synchronized >2014-07-31 06:24:22,423 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-64) Attempting to update VMs/Templates Ovf. >2014-07-31 07:23:55,261 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-63) Principal admin::internal synchronized >2014-07-31 07:24:22,460 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-85) Attempting to update VMs/Templates Ovf. >2014-07-31 07:59:17,969 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-47) domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 in problem. vds: rack >2014-07-31 07:59:17,971 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-47) domain 44559ec0-6bef-491c-bbd6-e67278f37987:export in problem. vds: rack >2014-07-31 08:00:56,471 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-66) [2734597c] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2014-07-31 08:00:56,475 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-66) [653adc38] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-07-31 08:00:56,479 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-66) [653adc38] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Active to Unknown, reason : null >2014-07-31 08:00:56,481 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-66) [653adc38] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Active to Unknown, reason : null >2014-07-31 08:00:56,495 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-66) [653adc38] Correlation ID: 653adc38, Call Stack: null, Custom Event ID: -1, Message: Invalid status on Data Center Default. Setting Data Center status to Non Responsive (On host rack, Error: Cannot access master storage domain). >2014-07-31 08:02:56,535 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-66) [653adc38] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2014-07-31 08:02:56,537 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-66) [653adc38] hostFromVds::selectedVds - rack, spmStatus returned null! >2014-07-31 08:02:56,538 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-66) [653adc38] IrsBroker::Failed::GetStoragePoolInfoVDS >2014-07-31 08:02:56,539 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-66) [653adc38] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2014-07-31 08:02:56,566 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-16) [7b3e6f1d] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage >2014-07-31 08:02:56,582 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-16) [7b3e6f1d] START, ResetIrsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, ignoreStopFailed = true), log id: 688a1691 >2014-07-31 08:02:56,585 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-16) [7b3e6f1d] START, SpmStopVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9), log id: 1e2def37 >2014-07-31 08:02:56,596 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-16) [7b3e6f1d] SpmStopVDSCommand::Stopping SPM on vds rack, pool id 00000002-0002-0002-0002-0000000001d9 >2014-07-31 08:04:56,516 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-52) Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2014-07-31 08:04:56,519 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-52) hostFromVds::selectedVds - rack, spmStatus returned null! >2014-07-31 08:04:56,520 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-52) IrsBroker::Failed::GetStoragePoolInfoVDS >2014-07-31 08:04:56,521 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-52) Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2014-07-31 08:05:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-7) [2b0f39f8] Autorecovering 1 storage domains >2014-07-31 08:05:00,005 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-7) [2b0f39f8] Autorecovering storage domains id: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 >2014-07-31 08:05:00,008 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-7) [42c2440e] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage >2014-07-31 08:05:00,010 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-7) [42c2440e] ConnectDomainToStorage. Before Connect all hosts to pool. Time:7/31/14 8:05 AM >2014-07-31 08:05:00,021 INFO [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-8-thread-8) [67959ed5] Running command: ConnectStorageToVdsCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2014-07-31 08:05:00,029 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-8) [67959ed5] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 4079a353 >2014-07-31 08:05:56,598 WARN [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-16) [7b3e6f1d] could not stop spm of pool 00000002-0002-0002-0002-0000000001d9 on vds 17f7fe56-5f32-40ef-b098-611b84484e0b - reason: org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcRunTimeException: java.util.concurrent.TimeoutException >2014-07-31 08:05:56,600 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-16) [7b3e6f1d] FINISH, SpmStopVDSCommand, log id: 1e2def37 >2014-07-31 08:05:56,605 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-16) [7b3e6f1d] FINISH, ResetIrsVDSCommand, log id: 688a1691 >2014-07-31 08:05:56,615 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-16) [7b3e6f1d] Correlation ID: 7b3e6f1d, Job ID: 8f6c7c49-c618-4c4b-aaf3-76afbb87ea8f, Call Stack: null, Custom Event ID: -1, Message: Failed to Reconstruct Master Domain for Data Center Default. >2014-07-31 08:05:56,620 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-2) [653adc38] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue >2014-07-31 08:05:56,622 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-21) starting processDomainRecovery for domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 >2014-07-31 08:05:56,686 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-21) Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 was reported by all hosts in status UP as problematic. Not moving the domain to NonOperational because it is being reconstructed now. >2014-07-31 08:05:56,730 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-21) [396d3c55] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage >2014-07-31 08:05:56,762 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-21) [396d3c55] Removing vds [] from the domain in maintenance cache >2014-07-31 08:05:56,763 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-21) [396d3c55] Removing host(s) [] from hosts unseen domain report cache >2014-07-31 08:05:56,764 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-2) [653adc38] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue >2014-07-31 08:05:56,766 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-19) starting processDomainRecovery for domain 44559ec0-6bef-491c-bbd6-e67278f37987:export >2014-07-31 08:05:56,771 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-19) Domain 44559ec0-6bef-491c-bbd6-e67278f37987:export was reported by all hosts in status UP as problematic. Moving the domain to NonOperational. >2014-07-31 08:05:56,785 INFO [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-19) [7f63a243] Lock Acquired to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE >, sharedLocks= ] >2014-07-31 08:05:56,794 INFO [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-19) [7f63a243] Running command: DeactivateStorageDomainCommand internal: true. Entities affected : ID: 44559ec0-6bef-491c-bbd6-e67278f37987 Type: Storage >2014-07-31 08:05:56,800 INFO [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-19) [7f63a243] Lock freed to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE >, sharedLocks= ] >2014-07-31 08:05:56,811 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-19) [7f63a243] Correlation ID: 7f63a243, Job ID: 55f328a5-b0e3-4d78-b4e4-b688ed2e2e20, Call Stack: null, Custom Event ID: -1, Message: Storage Domain export (Data Center Default) was deactivated by system because it's not visible by any of the hosts. >2014-07-31 08:05:56,818 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-19) [7f63a243] Removing vds [] from the domain in maintenance cache >2014-07-31 08:05:56,819 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-19) [7f63a243] Removing host(s) [] from hosts unseen domain report cache >2014-07-31 08:06:00,043 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-8) [67959ed5] FINISH, ConnectStorageServerVDSCommand, return: {5083f370-af09-457b-9b73-3440e8b65a07=0}, log id: 4079a353 >2014-07-31 08:06:00,046 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-7) [42c2440e] ConnectDomainToStorage. After Connect all hosts to pool. Time:7/31/14 8:06 AM >2014-07-31 08:06:56,534 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-6) [4494a716] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: VDSNetworkException: org.apache.commons.httpclient.NoHttpResponseException: The server 10.34.62.208 failed to respond >2014-07-31 08:06:56,537 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-6) [4494a716] hostFromVds::selectedVds - rack, spmStatus returned null! >2014-07-31 08:06:57,614 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ListVDSCommand] (DefaultQuartzScheduler_Worker-27) Command ListVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]) execution failed. Exception: VDSRecoveringException: Recovering from crash or Initializing >2014-07-31 08:06:57,616 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-27) Failed vds listing, vds = 17f7fe56-5f32-40ef-b098-611b84484e0b : rack, error = org.ovirt.engine.core.vdsbroker.vdsbroker.VDSRecoveringException: Recovering from crash or Initializing >2014-07-31 08:06:57,637 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-27) Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSRecoveringException: Recovering from crash or Initializing > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.vdsbroker.ListVDSCommand.executeVdsBrokerCommand(ListVDSCommand.java:24) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:406) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.fetchRunningVms(VdsUpdateRunTimeInfo.java:971) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVmStats(VdsUpdateRunTimeInfo.java:920) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:638) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:235) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Host rack is initializing. Message: Recovering from crash or Initializing >2014-07-31 08:06:57,648 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-27) Failed to refresh VDS , vds = 17f7fe56-5f32-40ef-b098-611b84484e0b : rack, error = Recovering from crash or Initializing, continuing. >2014-07-31 08:07:00,876 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-95) [ceef331] START, GetHardwareInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]), log id: 166134dd >2014-07-31 08:07:00,927 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-95) [ceef331] FINISH, GetHardwareInfoVDSCommand, log id: 166134dd >2014-07-31 08:07:00,982 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-95) [41f51a04] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS >2014-07-31 08:07:00,992 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-95) [41f51a04] Cancelling the recovery from crash timer for VDS 17f7fe56-5f32-40ef-b098-611b84484e0b because vds started initializing >2014-07-31 08:07:00,993 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-95) [13b4a74d] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-07-31 08:07:01,003 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-95) [33e7d6c4] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-07-31 08:07:01,010 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-95) [33e7d6c4] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, storageType = NFS, connectionList = [{ id: 05047c9a-152e-4ef2-b118-4a1753c1eae0, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35export, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 3a7ce43d >2014-07-31 08:07:01,077 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-95) [33e7d6c4] FINISH, ConnectStorageServerVDSCommand, return: {5083f370-af09-457b-9b73-3440e8b65a07=0, 05047c9a-152e-4ef2-b118-4a1753c1eae0=0}, log id: 3a7ce43d >2014-07-31 08:07:01,078 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-95) [33e7d6c4] Host rack storage connection was succeeded >2014-07-31 08:07:01,084 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-44) START, ConnectStoragePoolVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, masterVersion = 1), log id: 68808caa >2014-07-31 08:07:06,541 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-75) [52dee25c] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Inactive to Unknown, reason : no reporting hosts >2014-07-31 08:07:06,545 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-75) [52dee25c] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Inactive to Unknown, reason : no reporting hosts >2014-07-31 08:08:01,498 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-44) FINISH, ConnectStoragePoolVDSCommand, log id: 68808caa >2014-07-31 08:08:01,512 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-95) [33e7d6c4] START, SetMOMPolicyParametersVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 5a2647d >2014-07-31 08:08:01,605 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-95) [33e7d6c4] FINISH, SetMOMPolicyParametersVDSCommand, log id: 5a2647d >2014-07-31 08:08:01,613 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-95) [33e7d6c4] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host rack was set to Up. >2014-07-31 08:08:01,620 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-95) [33e7d6c4] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host rack. >2014-07-31 08:08:01,638 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-95) [4c7ac6a1] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS >2014-07-31 08:08:06,661 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-3) hostFromVds::selectedVds - rack, spmStatus Free, storage pool Default >2014-07-31 08:08:06,668 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-3) starting spm on vds rack, storage pool Default, prevId -1, LVER -1 >2014-07-31 08:08:06,670 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-3) START, SpmStartVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, prevId=-1, prevLVER=-1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 425c0c89 >2014-07-31 08:08:06,719 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-3) spmStart polling started: taskId = 4fd9b3ab-eccf-41fa-8086-a7eee492113b >2014-07-31 08:08:07,765 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-3) spmStart polling ended: taskId = 4fd9b3ab-eccf-41fa-8086-a7eee492113b task status = finished >2014-07-31 08:08:07,822 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-3) spmStart polling ended, spm status: SPM >2014-07-31 08:08:07,823 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-3) START, HSMClearTaskVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, taskId=4fd9b3ab-eccf-41fa-8086-a7eee492113b), log id: 3a621dfe >2014-07-31 08:08:07,869 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-3) FINISH, HSMClearTaskVDSCommand, log id: 3a621dfe >2014-07-31 08:08:07,870 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-3) FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@8dddd8e, log id: 425c0c89 >2014-07-31 08:08:07,874 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-3) Initialize Irs proxy from vds: 10.34.62.208 >2014-07-31 08:08:07,881 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-3) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208). >2014-07-31 08:08:07,882 WARN [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-8-thread-12) Executing a command: java.util.concurrent.FutureTask , but note that there are 1 tasks in the queue. >2014-07-31 08:08:07,895 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-12) START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 666f039b >2014-07-31 08:08:07,922 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-12) -- executeIrsBrokerCommand: Attempting on storage pool 00000002-0002-0002-0002-0000000001d9 >2014-07-31 08:08:07,924 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-12) START, HSMGetAllTasksInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 23472fec >2014-07-31 08:08:07,929 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-12) FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 23472fec >2014-07-31 08:08:07,930 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-12) FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 666f039b >2014-07-31 08:08:07,931 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (org.ovirt.thread.pool-8-thread-12) Discovered no tasks on Storage Pool Default >2014-07-31 08:23:55,267 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-91) Principal admin::internal synchronized >2014-07-31 08:24:22,475 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-93) [341aec49] Attempting to update VMs/Templates Ovf. >2014-07-31 08:38:03,596 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp--127.0.0.1-8702-11) Running command: LoginUserCommand internal: false. >2014-07-31 08:38:03,621 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-11) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-07-31 08:38:32,640 INFO [org.ovirt.engine.core.bll.LogoutUserCommand] (ajp--127.0.0.1-8702-5) [12abc774] Running command: LogoutUserCommand internal: false. >2014-07-31 08:38:32,646 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-5) [12abc774] Correlation ID: 12abc774, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-07-31 09:23:55,273 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-72) Principal admin::internal synchronized >2014-07-31 09:24:22,492 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-41) [5d8b5376] Attempting to update VMs/Templates Ovf. >2014-07-31 09:35:23,463 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp--127.0.0.1-8702-6) Running command: LoginUserCommand internal: false. >2014-07-31 09:35:23,469 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-07-31 09:35:23,470 ERROR [org.ovirt.engine.core.aaa.filters.RestApiSessionMgmtFilter] (ajp--127.0.0.1-8702-6) Session-TTL header was not passed. Not setting TTL value >2014-07-31 09:35:23,572 ERROR [org.ovirt.engine.core.aaa.filters.RestApiSessionMgmtFilter] (ajp--127.0.0.1-8702-7) Session-TTL header was not passed. Not setting TTL value >2014-07-31 09:44:20,481 WARN [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-8-thread-1) Executing a command: java.util.concurrent.FutureTask , but note that there are 1 tasks in the queue. >2014-07-31 10:23:55,281 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-65) Principal admin::internal synchronized >2014-07-31 10:24:22,509 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-100) [52a75f18] Attempting to update VMs/Templates Ovf. >2014-07-31 11:07:45,664 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-14) domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 in problem. vds: rack >2014-07-31 11:08:00,785 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-19) domain 44559ec0-6bef-491c-bbd6-e67278f37987:export in problem. vds: rack >2014-07-31 11:09:24,857 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-6) [4494a716] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2014-07-31 11:09:24,863 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-6) [7f1e369e] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-07-31 11:09:24,866 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-6) [7f1e369e] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Active to Unknown, reason : null >2014-07-31 11:09:24,868 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-6) [7f1e369e] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Active to Unknown, reason : null >2014-07-31 11:09:24,881 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-6) [7f1e369e] Correlation ID: 7f1e369e, Call Stack: null, Custom Event ID: -1, Message: Invalid status on Data Center Default. Setting Data Center status to Non Responsive (On host rack, Error: Cannot access master storage domain). >2014-07-31 11:11:24,911 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-6) [7f1e369e] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2014-07-31 11:11:24,913 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-6) [7f1e369e] hostFromVds::selectedVds - rack, spmStatus returned null! >2014-07-31 11:11:24,914 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-6) [7f1e369e] IrsBroker::Failed::GetStoragePoolInfoVDS >2014-07-31 11:11:24,915 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-6) [7f1e369e] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2014-07-31 11:11:24,951 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-12) [385b441c] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage >2014-07-31 11:11:24,977 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-12) [385b441c] START, ResetIrsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, ignoreStopFailed = true), log id: 5b2f2794 >2014-07-31 11:11:24,981 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-12) [385b441c] START, SpmStopVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9), log id: 590dc58f >2014-07-31 11:11:24,987 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-12) [385b441c] SpmStopVDSCommand::Stopping SPM on vds rack, pool id 00000002-0002-0002-0002-0000000001d9 >2014-07-31 11:13:24,881 WARN [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-12) [385b441c] could not stop spm of pool 00000002-0002-0002-0002-0000000001d9 on vds 17f7fe56-5f32-40ef-b098-611b84484e0b - reason: org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcRunTimeException: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException >2014-07-31 11:13:24,885 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ListVDSCommand] (DefaultQuartzScheduler_Worker-3) Command ListVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-07-31 11:13:24,887 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-3) Host rack is not responding. It will stay in Connecting state for a grace period of $80 seconds and after that an attempt to fence the host will be issued. >2014-07-31 11:13:24,888 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-12) [385b441c] FINISH, SpmStopVDSCommand, log id: 590dc58f >2014-07-31 11:13:24,890 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-32) Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: VDSNetworkException: org.apache.commons.httpclient.NoHttpResponseException: The server 10.34.62.208 failed to respond >2014-07-31 11:13:24,894 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-32) hostFromVds::selectedVds - rack, spmStatus returned null! >2014-07-31 11:13:24,898 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-12) [385b441c] FINISH, ResetIrsVDSCommand, log id: 5b2f2794 >2014-07-31 11:13:24,888 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-3) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:406) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.fetchRunningVms(VdsUpdateRunTimeInfo.java:971) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVmStats(VdsUpdateRunTimeInfo.java:920) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:638) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:235) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) [:1.7.0_55] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_55] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_55] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_55] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_55] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_55] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_55] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_55] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy82.list(Unknown Source) > at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source) [:1.7.0_55] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_55] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_55] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_55] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_55] > >2014-07-31 11:13:24,908 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-12) [385b441c] Correlation ID: 385b441c, Job ID: a6b4d423-858c-4f8d-94f8-b2e030d5a254, Call Stack: null, Custom Event ID: -1, Message: Failed to Reconstruct Master Domain for Data Center Default. >2014-07-31 11:13:24,934 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-3) Failed to refresh VDS , vds = 17f7fe56-5f32-40ef-b098-611b84484e0b : rack, VDS Network Error, continuing. >java.net.ConnectException: Connection refused >2014-07-31 11:13:24,935 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-9) [7f1e369e] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue >2014-07-31 11:13:24,938 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-48) starting processDomainRecovery for domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 >2014-07-31 11:13:25,006 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-48) Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 was reported by all hosts in status UP as problematic. Not moving the domain to NonOperational because it is being reconstructed now. >2014-07-31 11:13:25,028 WARN [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-48) [2015a55d] CanDoAction of action ReconstructMasterDomain failed. Reasons:VAR__ACTION__RECONSTRUCT_MASTER,VAR__TYPE__STORAGE__DOMAIN,ACTION_TYPE_FAILED_NO_VDS_IN_POOL >2014-07-31 11:13:25,035 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-48) [2015a55d] Removing vds [] from the domain in maintenance cache >2014-07-31 11:13:25,036 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-48) [2015a55d] Removing host(s) [] from hosts unseen domain report cache >2014-07-31 11:13:25,037 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-9) [7f1e369e] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue >2014-07-31 11:13:25,039 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-43) starting processDomainRecovery for domain 44559ec0-6bef-491c-bbd6-e67278f37987:export >2014-07-31 11:13:25,044 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-43) Domain 44559ec0-6bef-491c-bbd6-e67278f37987:export was reported by all hosts in status UP as problematic. Moving the domain to NonOperational. >2014-07-31 11:13:25,060 INFO [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-43) [20b596ad] Lock Acquired to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE >, sharedLocks= ] >2014-07-31 11:13:25,067 INFO [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-43) [20b596ad] Running command: DeactivateStorageDomainCommand internal: true. Entities affected : ID: 44559ec0-6bef-491c-bbd6-e67278f37987 Type: Storage >2014-07-31 11:13:25,074 INFO [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-43) [20b596ad] Lock freed to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE >, sharedLocks= ] >2014-07-31 11:13:25,084 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-43) [20b596ad] Correlation ID: 20b596ad, Job ID: 56a226b7-1cb3-4ce0-bb3e-8dfde4b86066, Call Stack: null, Custom Event ID: -1, Message: Storage Domain export (Data Center Default) was deactivated by system because it's not visible by any of the hosts. >2014-07-31 11:13:25,090 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-43) [20b596ad] Removing vds [] from the domain in maintenance cache >2014-07-31 11:13:25,090 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-43) [20b596ad] Removing host(s) [] from hosts unseen domain report cache >2014-07-31 11:13:28,197 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-68) [6dc0d36f] START, GetHardwareInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]), log id: 108e0e8d >2014-07-31 11:13:28,247 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-68) [6dc0d36f] FINISH, GetHardwareInfoVDSCommand, log id: 108e0e8d >2014-07-31 11:13:28,276 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-68) [11c9633d] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS >2014-07-31 11:13:28,327 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-68) [2c688054] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-07-31 11:13:28,334 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-68) [2d96ea92] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-07-31 11:13:28,340 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-68) [2d96ea92] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, storageType = NFS, connectionList = [{ id: 05047c9a-152e-4ef2-b118-4a1753c1eae0, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35export, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: fe431fb >2014-07-31 11:13:28,402 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-68) [2d96ea92] FINISH, ConnectStorageServerVDSCommand, return: {5083f370-af09-457b-9b73-3440e8b65a07=0, 05047c9a-152e-4ef2-b118-4a1753c1eae0=0}, log id: fe431fb >2014-07-31 11:13:28,403 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-68) [2d96ea92] Host rack storage connection was succeeded >2014-07-31 11:13:28,409 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-39) START, ConnectStoragePoolVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, masterVersion = 1), log id: 2e70b61b >2014-07-31 11:13:34,902 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-69) [52d379dc] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Inactive to Unknown, reason : no reporting hosts >2014-07-31 11:13:34,905 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-69) [52d379dc] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Inactive to Unknown, reason : no reporting hosts >2014-07-31 11:14:28,817 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-39) FINISH, ConnectStoragePoolVDSCommand, log id: 2e70b61b >2014-07-31 11:14:28,830 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-68) [2d96ea92] START, SetMOMPolicyParametersVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 2a421bf9 >2014-07-31 11:14:28,923 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-68) [2d96ea92] FINISH, SetMOMPolicyParametersVDSCommand, log id: 2a421bf9 >2014-07-31 11:14:28,931 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-68) [2d96ea92] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host rack was set to Up. >2014-07-31 11:14:28,937 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-68) [2d96ea92] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host rack. >2014-07-31 11:14:28,953 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-68) [4341002] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS >2014-07-31 11:14:35,051 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-58) [7b19edd8] hostFromVds::selectedVds - rack, spmStatus Free, storage pool Default >2014-07-31 11:14:35,056 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-58) [7b19edd8] starting spm on vds rack, storage pool Default, prevId -1, LVER -1 >2014-07-31 11:14:35,059 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-58) [7b19edd8] START, SpmStartVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, prevId=-1, prevLVER=-1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 126eb4db >2014-07-31 11:14:35,108 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-58) [7b19edd8] spmStart polling started: taskId = 84af9baf-f8cc-4ea7-bd5a-a859f5023d6e >2014-07-31 11:14:36,114 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-58) [7b19edd8] spmStart polling ended: taskId = 84af9baf-f8cc-4ea7-bd5a-a859f5023d6e task status = finished >2014-07-31 11:14:36,170 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-58) [7b19edd8] spmStart polling ended, spm status: SPM >2014-07-31 11:14:36,172 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-58) [7b19edd8] START, HSMClearTaskVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, taskId=84af9baf-f8cc-4ea7-bd5a-a859f5023d6e), log id: 2acc59e7 >2014-07-31 11:14:36,217 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-58) [7b19edd8] FINISH, HSMClearTaskVDSCommand, log id: 2acc59e7 >2014-07-31 11:14:36,218 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-58) [7b19edd8] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@13b1b5ab, log id: 126eb4db >2014-07-31 11:14:36,223 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-58) [7b19edd8] Initialize Irs proxy from vds: 10.34.62.208 >2014-07-31 11:14:36,229 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-58) [7b19edd8] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208). >2014-07-31 11:14:36,239 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-13) [7b19edd8] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 41485ee4 >2014-07-31 11:14:36,274 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-13) [7b19edd8] -- executeIrsBrokerCommand: Attempting on storage pool 00000002-0002-0002-0002-0000000001d9 >2014-07-31 11:14:36,277 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-13) [7b19edd8] START, HSMGetAllTasksInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 55a33e6d >2014-07-31 11:14:36,282 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-13) [7b19edd8] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 55a33e6d >2014-07-31 11:14:36,283 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-13) [7b19edd8] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 41485ee4 >2014-07-31 11:14:36,284 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (org.ovirt.thread.pool-8-thread-13) [7b19edd8] Discovered no tasks on Storage Pool Default >2014-07-31 11:23:55,288 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-93) [341aec49] Principal admin::internal synchronized >2014-07-31 11:24:22,523 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-24) Attempting to update VMs/Templates Ovf. >2014-07-31 12:23:55,297 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-67) [2f0280a3] Principal admin::internal synchronized >2014-07-31 12:24:22,539 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-71) Attempting to update VMs/Templates Ovf. >2014-07-31 13:23:55,302 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-77) [79b9f7da] Principal admin::internal synchronized >2014-07-31 13:24:22,563 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-75) [52dee25c] Attempting to update VMs/Templates Ovf. >2014-07-31 14:14:19,361 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-36) domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 in problem. vds: rack >2014-07-31 14:14:19,364 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-36) domain 44559ec0-6bef-491c-bbd6-e67278f37987:export in problem. vds: rack >2014-07-31 14:15:53,449 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-64) Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2014-07-31 14:15:53,456 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-64) [686e64a1] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-07-31 14:15:53,460 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-64) [686e64a1] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Active to Unknown, reason : null >2014-07-31 14:15:53,462 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-64) [686e64a1] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Active to Unknown, reason : null >2014-07-31 14:15:53,475 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-64) [686e64a1] Correlation ID: 686e64a1, Call Stack: null, Custom Event ID: -1, Message: Invalid status on Data Center Default. Setting Data Center status to Non Responsive (On host rack, Error: Cannot access master storage domain). >2014-07-31 14:17:53,502 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-64) [686e64a1] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2014-07-31 14:17:53,504 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-64) [686e64a1] hostFromVds::selectedVds - rack, spmStatus returned null! >2014-07-31 14:17:53,505 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-64) [686e64a1] IrsBroker::Failed::GetStoragePoolInfoVDS >2014-07-31 14:17:53,506 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-64) [686e64a1] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2014-07-31 14:17:53,553 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-24) [cd347f3] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage >2014-07-31 14:17:53,571 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-24) [cd347f3] START, ResetIrsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, ignoreStopFailed = true), log id: 7b586e56 >2014-07-31 14:17:53,575 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-24) [cd347f3] START, SpmStopVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9), log id: 190eaae0 >2014-07-31 14:17:53,581 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-24) [cd347f3] SpmStopVDSCommand::Stopping SPM on vds rack, pool id 00000002-0002-0002-0002-0000000001d9 >2014-07-31 14:19:53,473 WARN [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-24) [cd347f3] could not stop spm of pool 00000002-0002-0002-0002-0000000001d9 on vds 17f7fe56-5f32-40ef-b098-611b84484e0b - reason: org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcRunTimeException: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException >2014-07-31 14:19:53,476 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-24) [cd347f3] FINISH, SpmStopVDSCommand, log id: 190eaae0 >2014-07-31 14:19:53,475 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-34) [5484a050] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: VDSNetworkException: org.apache.commons.httpclient.NoHttpResponseException: The server 10.34.62.208 failed to respond >2014-07-31 14:19:53,477 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetStatsVDSCommand] (DefaultQuartzScheduler_Worker-55) [296dca52] Command GetStatsVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-07-31 14:19:53,480 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-34) [5484a050] hostFromVds::selectedVds - rack, spmStatus returned null! >2014-07-31 14:19:53,486 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-55) [296dca52] Host rack is not responding. It will stay in Connecting state for a grace period of $80 seconds and after that an attempt to fence the host will be issued. >2014-07-31 14:19:53,487 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-55) [296dca52] vds::refreshVdsStats Failed getVdsStats, vds = 17f7fe56-5f32-40ef-b098-611b84484e0b : rack, error = VDSNetworkException: java.net.ConnectException: Connection refused >2014-07-31 14:19:53,489 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-55) [296dca52] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsStats(VdsUpdateRunTimeInfo.java:669) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:623) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:235) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) [:1.7.0_55] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_55] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_55] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_55] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_55] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_55] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_55] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_55] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy82.getVdsStats(Unknown Source) > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_55] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_55] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_55] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_55] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_55] > >2014-07-31 14:19:53,494 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-24) [cd347f3] FINISH, ResetIrsVDSCommand, log id: 7b586e56 >2014-07-31 14:19:53,522 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-55) [296dca52] Failed to refresh VDS , vds = 17f7fe56-5f32-40ef-b098-611b84484e0b : rack, VDS Network Error, continuing. >java.net.ConnectException: Connection refused >2014-07-31 14:19:53,526 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-24) [cd347f3] Correlation ID: cd347f3, Job ID: b7387082-f3da-4de5-8add-f976dc51cf8e, Call Stack: null, Custom Event ID: -1, Message: Failed to Reconstruct Master Domain for Data Center Default. >2014-07-31 14:19:53,532 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-50) [686e64a1] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue >2014-07-31 14:19:53,534 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-12) starting processDomainRecovery for domain 44559ec0-6bef-491c-bbd6-e67278f37987:export >2014-07-31 14:19:53,588 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-12) Domain 44559ec0-6bef-491c-bbd6-e67278f37987:export was reported by all hosts in status UP as problematic. Moving the domain to NonOperational. >2014-07-31 14:19:53,605 INFO [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-12) [5448ed13] Lock Acquired to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE >, sharedLocks= ] >2014-07-31 14:19:53,611 INFO [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-12) [5448ed13] Running command: DeactivateStorageDomainCommand internal: true. Entities affected : ID: 44559ec0-6bef-491c-bbd6-e67278f37987 Type: Storage >2014-07-31 14:19:53,616 INFO [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-12) [5448ed13] Lock freed to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE >, sharedLocks= ] >2014-07-31 14:19:53,624 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-12) [5448ed13] Correlation ID: 5448ed13, Job ID: adaec078-f569-4930-b790-6fbfbf7e4a6e, Call Stack: null, Custom Event ID: -1, Message: Storage Domain export (Data Center Default) was deactivated by system because it's not visible by any of the hosts. >2014-07-31 14:19:53,629 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-12) [5448ed13] Removing vds [] from the domain in maintenance cache >2014-07-31 14:19:53,630 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-12) [5448ed13] Removing host(s) [] from hosts unseen domain report cache >2014-07-31 14:19:53,632 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-19) starting processDomainRecovery for domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 >2014-07-31 14:19:53,636 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-19) Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 was reported by all hosts in status UP as problematic. Not moving the domain to NonOperational because it is being reconstructed now. >2014-07-31 14:19:53,655 WARN [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-19) [63924f71] CanDoAction of action ReconstructMasterDomain failed. Reasons:VAR__ACTION__RECONSTRUCT_MASTER,VAR__TYPE__STORAGE__DOMAIN,ACTION_TYPE_FAILED_NO_VDS_IN_POOL >2014-07-31 14:19:53,662 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-19) [63924f71] Removing vds [] from the domain in maintenance cache >2014-07-31 14:19:53,663 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-19) [63924f71] Removing host(s) [] from hosts unseen domain report cache >2014-07-31 14:19:53,664 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-50) [686e64a1] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue >2014-07-31 14:19:56,769 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-42) START, GetHardwareInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]), log id: 485a9c02 >2014-07-31 14:19:56,830 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-42) FINISH, GetHardwareInfoVDSCommand, log id: 485a9c02 >2014-07-31 14:19:56,856 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-42) [7eb60aa4] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS >2014-07-31 14:19:56,906 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-42) [3585d0b5] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-07-31 14:19:56,914 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-42) [1e1598cd] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-07-31 14:19:56,926 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-42) [1e1598cd] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, storageType = NFS, connectionList = [{ id: 05047c9a-152e-4ef2-b118-4a1753c1eae0, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35export, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 7b720444 >2014-07-31 14:19:56,987 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-42) [1e1598cd] FINISH, ConnectStorageServerVDSCommand, return: {5083f370-af09-457b-9b73-3440e8b65a07=0, 05047c9a-152e-4ef2-b118-4a1753c1eae0=0}, log id: 7b720444 >2014-07-31 14:19:56,989 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-42) [1e1598cd] Host rack storage connection was succeeded >2014-07-31 14:19:56,998 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-2) START, ConnectStoragePoolVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, masterVersion = 1), log id: 33616302 >2014-07-31 14:20:00,005 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-85) Autorecovering 2 storage domains >2014-07-31 14:20:00,006 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-85) Autorecovering storage domains id: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 >2014-07-31 14:20:00,008 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-85) [7390d6d6] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage >2014-07-31 14:20:00,009 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-85) [7390d6d6] ConnectDomainToStorage. Before Connect all hosts to pool. Time:7/31/14 2:20 PM >2014-07-31 14:20:00,014 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-85) [7390d6d6] ConnectDomainToStorage. After Connect all hosts to pool. Time:7/31/14 2:20 PM >2014-07-31 14:20:00,016 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-85) [7390d6d6] Autorecovering storage domains id: 44559ec0-6bef-491c-bbd6-e67278f37987 >2014-07-31 14:20:00,018 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-85) [6a1682ec] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 44559ec0-6bef-491c-bbd6-e67278f37987 Type: Storage >2014-07-31 14:20:00,019 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-85) [6a1682ec] ConnectDomainToStorage. Before Connect all hosts to pool. Time:7/31/14 2:20 PM >2014-07-31 14:20:00,024 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-85) [6a1682ec] ConnectDomainToStorage. After Connect all hosts to pool. Time:7/31/14 2:20 PM >2014-07-31 14:20:03,487 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-24) Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Inactive to Unknown, reason : no reporting hosts >2014-07-31 14:20:03,491 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-24) Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Inactive to Unknown, reason : no reporting hosts >2014-07-31 14:20:57,377 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-2) FINISH, ConnectStoragePoolVDSCommand, log id: 33616302 >2014-07-31 14:20:57,392 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-42) [1e1598cd] START, SetMOMPolicyParametersVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: d100a7a >2014-07-31 14:20:57,485 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-42) [1e1598cd] FINISH, SetMOMPolicyParametersVDSCommand, log id: d100a7a >2014-07-31 14:20:57,493 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-42) [1e1598cd] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host rack was set to Up. >2014-07-31 14:20:57,499 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-42) [1e1598cd] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host rack. >2014-07-31 14:20:57,517 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-42) [3eb4f572] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS >2014-07-31 14:21:03,613 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-94) hostFromVds::selectedVds - rack, spmStatus Free, storage pool Default >2014-07-31 14:21:03,618 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-94) starting spm on vds rack, storage pool Default, prevId -1, LVER -1 >2014-07-31 14:21:03,621 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-94) START, SpmStartVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, prevId=-1, prevLVER=-1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 35f8090e >2014-07-31 14:21:03,668 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-94) spmStart polling started: taskId = fbecd537-eb0e-42d0-aa4b-0711a325814f >2014-07-31 14:21:04,674 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-94) spmStart polling ended: taskId = fbecd537-eb0e-42d0-aa4b-0711a325814f task status = finished >2014-07-31 14:21:04,730 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-94) spmStart polling ended, spm status: SPM >2014-07-31 14:21:04,733 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-94) START, HSMClearTaskVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, taskId=fbecd537-eb0e-42d0-aa4b-0711a325814f), log id: 2ba224ca >2014-07-31 14:21:04,777 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-94) FINISH, HSMClearTaskVDSCommand, log id: 2ba224ca >2014-07-31 14:21:04,778 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-94) FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@1d28b13d, log id: 35f8090e >2014-07-31 14:21:04,784 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-94) Initialize Irs proxy from vds: 10.34.62.208 >2014-07-31 14:21:04,791 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-94) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208). >2014-07-31 14:21:04,792 WARN [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-8-thread-28) Executing a command: java.util.concurrent.FutureTask , but note that there are 1 tasks in the queue. >2014-07-31 14:21:04,807 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-28) START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 2edfc7aa >2014-07-31 14:21:04,831 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-28) -- executeIrsBrokerCommand: Attempting on storage pool 00000002-0002-0002-0002-0000000001d9 >2014-07-31 14:21:04,833 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-28) START, HSMGetAllTasksInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 44cb155e >2014-07-31 14:21:04,838 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-28) FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 44cb155e >2014-07-31 14:21:04,839 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-28) FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 2edfc7aa >2014-07-31 14:21:04,840 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (org.ovirt.thread.pool-8-thread-28) Discovered no tasks on Storage Pool Default >2014-07-31 14:23:55,310 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-15) Principal admin::internal synchronized >2014-07-31 14:24:22,578 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-65) Attempting to update VMs/Templates Ovf. >2014-07-31 15:23:55,316 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-20) [3adf532b] Principal admin::internal synchronized >2014-07-31 15:24:22,595 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-50) [375fcc6] Attempting to update VMs/Templates Ovf. >2014-07-31 16:23:55,321 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-42) [3eb4f572] Principal admin::internal synchronized >2014-07-31 16:24:22,610 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-61) [58bfe349] Attempting to update VMs/Templates Ovf. >2014-07-31 17:20:51,989 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-10) domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 in problem. vds: rack >2014-07-31 17:21:07,108 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-6) domain 44559ec0-6bef-491c-bbd6-e67278f37987:export in problem. vds: rack >2014-07-31 17:22:28,454 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-85) [6a1682ec] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2014-07-31 17:22:28,458 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-85) [5e890dae] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-07-31 17:22:28,461 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-85) [5e890dae] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Active to Unknown, reason : null >2014-07-31 17:22:28,463 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-85) [5e890dae] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Active to Unknown, reason : null >2014-07-31 17:22:28,475 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-85) [5e890dae] Correlation ID: 5e890dae, Call Stack: null, Custom Event ID: -1, Message: Invalid status on Data Center Default. Setting Data Center status to Non Responsive (On host rack, Error: Cannot access master storage domain). >2014-07-31 17:23:55,327 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-21) Principal admin::internal synchronized >2014-07-31 17:24:22,638 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-99) Attempting to update VMs/Templates Ovf. >2014-07-31 17:24:28,493 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-85) [5e890dae] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2014-07-31 17:24:28,495 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-85) [5e890dae] hostFromVds::selectedVds - rack, spmStatus returned null! >2014-07-31 17:24:28,496 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-85) [5e890dae] IrsBroker::Failed::GetStoragePoolInfoVDS >2014-07-31 17:24:28,497 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-85) [5e890dae] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2014-07-31 17:24:28,544 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-9) [338c4d71] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage >2014-07-31 17:24:28,561 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-9) [338c4d71] START, ResetIrsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, ignoreStopFailed = true), log id: 5a64bf32 >2014-07-31 17:24:28,564 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-9) [338c4d71] START, SpmStopVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9), log id: 394d37e4 >2014-07-31 17:24:28,570 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-9) [338c4d71] SpmStopVDSCommand::Stopping SPM on vds rack, pool id 00000002-0002-0002-0002-0000000001d9 >2014-07-31 17:25:00,005 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-55) [296dca52] Autorecovering 1 storage domains >2014-07-31 17:25:00,006 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-55) [296dca52] Autorecovering storage domains id: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 >2014-07-31 17:25:00,008 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-55) [7526d482] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage >2014-07-31 17:25:00,009 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-55) [7526d482] ConnectDomainToStorage. Before Connect all hosts to pool. Time:7/31/14 5:25 PM >2014-07-31 17:25:00,019 INFO [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-8-thread-4) [34c0b244] Running command: ConnectStorageToVdsCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2014-07-31 17:25:00,032 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-4) [34c0b244] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 5467fea1 >2014-07-31 17:26:28,462 WARN [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-9) [338c4d71] could not stop spm of pool 00000002-0002-0002-0002-0000000001d9 on vds 17f7fe56-5f32-40ef-b098-611b84484e0b - reason: org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcRunTimeException: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException >2014-07-31 17:26:28,466 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-9) [338c4d71] FINISH, SpmStopVDSCommand, log id: 394d37e4 >2014-07-31 17:26:28,473 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-4) [34c0b244] Command ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-07-31 17:26:28,475 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-4) [34c0b244] FINISH, ConnectStorageServerVDSCommand, log id: 5467fea1 >2014-07-31 17:26:28,476 ERROR [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-8-thread-4) [34c0b244] Command org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-07-31 17:26:28,479 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-88) Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: VDSNetworkException: org.apache.commons.httpclient.NoHttpResponseException: The server 10.34.62.208 failed to respond >2014-07-31 17:26:28,481 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-88) hostFromVds::selectedVds - rack, spmStatus returned null! >2014-07-31 17:26:28,479 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetStatsVDSCommand] (DefaultQuartzScheduler_Worker-21) Command GetStatsVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-07-31 17:26:28,485 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-21) Host rack is not responding. It will stay in Connecting state for a grace period of $80 seconds and after that an attempt to fence the host will be issued. >2014-07-31 17:26:28,486 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-21) vds::refreshVdsStats Failed getVdsStats, vds = 17f7fe56-5f32-40ef-b098-611b84484e0b : rack, error = VDSNetworkException: java.net.ConnectException: Connection refused >2014-07-31 17:26:28,488 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-21) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsStats(VdsUpdateRunTimeInfo.java:669) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:623) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:235) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) [:1.7.0_55] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_55] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_55] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_55] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_55] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_55] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_55] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_55] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy82.getVdsStats(Unknown Source) > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_55] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_55] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_55] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_55] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_55] > >2014-07-31 17:26:28,495 ERROR [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-8-thread-4) [34c0b244] Transaction rolled-back for command: org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand. >2014-07-31 17:26:28,491 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-9) [338c4d71] FINISH, ResetIrsVDSCommand, log id: 5a64bf32 >2014-07-31 17:26:28,518 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-55) [7526d482] ConnectDomainToStorage. After Connect all hosts to pool. Time:7/31/14 5:26 PM >2014-07-31 17:26:28,525 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-21) Failed to refresh VDS , vds = 17f7fe56-5f32-40ef-b098-611b84484e0b : rack, VDS Network Error, continuing. >java.net.ConnectException: Connection refused >2014-07-31 17:26:28,529 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-9) [338c4d71] Correlation ID: 338c4d71, Job ID: b98a97ea-2aed-4bc8-97b9-9617c02701bd, Call Stack: null, Custom Event ID: -1, Message: Failed to Reconstruct Master Domain for Data Center Default. >2014-07-31 17:26:28,534 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-38) [5e890dae] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue >2014-07-31 17:26:28,537 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-26) starting processDomainRecovery for domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 >2014-07-31 17:26:28,600 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-26) Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 was reported by all hosts in status UP as problematic. Not moving the domain to NonOperational because it is being reconstructed now. >2014-07-31 17:26:28,643 WARN [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-26) [357adb45] CanDoAction of action ReconstructMasterDomain failed. Reasons:VAR__ACTION__RECONSTRUCT_MASTER,VAR__TYPE__STORAGE__DOMAIN,ACTION_TYPE_FAILED_NO_VDS_IN_POOL >2014-07-31 17:26:28,651 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-26) [357adb45] Removing vds [] from the domain in maintenance cache >2014-07-31 17:26:28,652 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-26) [357adb45] Removing host(s) [] from hosts unseen domain report cache >2014-07-31 17:26:28,653 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-38) [5e890dae] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue >2014-07-31 17:26:28,656 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-25) starting processDomainRecovery for domain 44559ec0-6bef-491c-bbd6-e67278f37987:export >2014-07-31 17:26:28,660 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-25) Domain 44559ec0-6bef-491c-bbd6-e67278f37987:export was reported by all hosts in status UP as problematic. Moving the domain to NonOperational. >2014-07-31 17:26:28,675 INFO [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-25) [6e389c5f] Lock Acquired to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE >, sharedLocks= ] >2014-07-31 17:26:28,681 INFO [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-25) [6e389c5f] Running command: DeactivateStorageDomainCommand internal: true. Entities affected : ID: 44559ec0-6bef-491c-bbd6-e67278f37987 Type: Storage >2014-07-31 17:26:28,688 INFO [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-25) [6e389c5f] Lock freed to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE >, sharedLocks= ] >2014-07-31 17:26:28,697 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-25) [6e389c5f] Correlation ID: 6e389c5f, Job ID: 4c6d13de-dacf-4ca0-80e2-15476848fd76, Call Stack: null, Custom Event ID: -1, Message: Storage Domain export (Data Center Default) was deactivated by system because it's not visible by any of the hosts. >2014-07-31 17:26:28,702 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-25) [6e389c5f] Removing vds [] from the domain in maintenance cache >2014-07-31 17:26:28,703 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-25) [6e389c5f] Removing host(s) [] from hosts unseen domain report cache >2014-07-31 17:26:31,787 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-71) START, GetHardwareInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]), log id: 2b3d491f >2014-07-31 17:26:31,838 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-71) FINISH, GetHardwareInfoVDSCommand, log id: 2b3d491f >2014-07-31 17:26:31,865 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-71) [6bb078c0] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS >2014-07-31 17:26:31,917 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-71) [6e042952] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-07-31 17:26:31,924 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-71) [4763ce07] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-07-31 17:26:31,930 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-71) [4763ce07] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, storageType = NFS, connectionList = [{ id: 05047c9a-152e-4ef2-b118-4a1753c1eae0, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35export, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 45b9546b >2014-07-31 17:26:31,994 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-71) [4763ce07] FINISH, ConnectStorageServerVDSCommand, return: {5083f370-af09-457b-9b73-3440e8b65a07=0, 05047c9a-152e-4ef2-b118-4a1753c1eae0=0}, log id: 45b9546b >2014-07-31 17:26:31,996 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-71) [4763ce07] Host rack storage connection was succeeded >2014-07-31 17:26:32,002 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-32) START, ConnectStoragePoolVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, masterVersion = 1), log id: 672731ca >2014-07-31 17:26:38,485 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-57) [cd4c3ce] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Inactive to Unknown, reason : no reporting hosts >2014-07-31 17:26:38,489 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-57) [cd4c3ce] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Inactive to Unknown, reason : no reporting hosts >2014-07-31 17:27:32,570 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-32) FINISH, ConnectStoragePoolVDSCommand, log id: 672731ca >2014-07-31 17:27:32,582 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-71) [4763ce07] START, SetMOMPolicyParametersVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 736ea1da >2014-07-31 17:27:32,677 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-71) [4763ce07] FINISH, SetMOMPolicyParametersVDSCommand, log id: 736ea1da >2014-07-31 17:27:32,690 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-71) [4763ce07] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host rack was set to Up. >2014-07-31 17:27:32,697 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-71) [4763ce07] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host rack. >2014-07-31 17:27:32,712 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-71) [6cd4f56d] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS >2014-07-31 17:27:38,612 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-5) hostFromVds::selectedVds - rack, spmStatus Free, storage pool Default >2014-07-31 17:27:38,617 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-5) starting spm on vds rack, storage pool Default, prevId -1, LVER -1 >2014-07-31 17:27:38,619 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-5) START, SpmStartVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, prevId=-1, prevLVER=-1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 72b227e4 >2014-07-31 17:27:38,667 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-5) spmStart polling started: taskId = ed8dfde7-0eef-42b2-a89d-1cf15eed5898 >2014-07-31 17:27:39,674 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-5) spmStart polling ended: taskId = ed8dfde7-0eef-42b2-a89d-1cf15eed5898 task status = finished >2014-07-31 17:27:39,758 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-5) spmStart polling ended, spm status: SPM >2014-07-31 17:27:39,761 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-5) START, HSMClearTaskVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, taskId=ed8dfde7-0eef-42b2-a89d-1cf15eed5898), log id: 63ebc3a9 >2014-07-31 17:27:39,806 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-5) FINISH, HSMClearTaskVDSCommand, log id: 63ebc3a9 >2014-07-31 17:27:39,808 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-5) FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@402b2f44, log id: 72b227e4 >2014-07-31 17:27:39,812 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-5) Initialize Irs proxy from vds: 10.34.62.208 >2014-07-31 17:27:39,819 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-5) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208). >2014-07-31 17:27:39,820 WARN [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-8-thread-16) Executing a command: java.util.concurrent.FutureTask , but note that there are 1 tasks in the queue. >2014-07-31 17:27:39,838 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-16) START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 1d4b4cef >2014-07-31 17:27:39,889 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-16) -- executeIrsBrokerCommand: Attempting on storage pool 00000002-0002-0002-0002-0000000001d9 >2014-07-31 17:27:39,891 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-16) START, HSMGetAllTasksInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 3c2caa8c >2014-07-31 17:27:39,896 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-16) FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 3c2caa8c >2014-07-31 17:27:39,897 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-16) FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 1d4b4cef >2014-07-31 17:27:39,898 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (org.ovirt.thread.pool-8-thread-16) Discovered no tasks on Storage Pool Default >2014-07-31 18:23:55,335 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-49) Principal admin::internal synchronized >2014-07-31 18:24:22,640 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-62) [2e742a17] Attempting to update VMs/Templates Ovf. >2014-07-31 19:23:55,342 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-54) Principal admin::internal synchronized >2014-07-31 19:24:22,681 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-55) [7526d482] Attempting to update VMs/Templates Ovf. >2014-07-31 20:23:55,348 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-14) Principal admin::internal synchronized >2014-07-31 20:24:22,696 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-79) Attempting to update VMs/Templates Ovf. >2014-07-31 20:27:27,645 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-37) domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 in problem. vds: rack >2014-07-31 20:27:27,647 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-37) domain 44559ec0-6bef-491c-bbd6-e67278f37987:export in problem. vds: rack >2014-07-31 20:28:56,872 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-60) Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2014-07-31 20:28:56,878 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-60) [1cb6c42c] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-07-31 20:28:56,882 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-60) [1cb6c42c] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Active to Unknown, reason : null >2014-07-31 20:28:56,885 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-60) [1cb6c42c] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Active to Unknown, reason : null >2014-07-31 20:28:56,901 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-60) [1cb6c42c] Correlation ID: 1cb6c42c, Call Stack: null, Custom Event ID: -1, Message: Invalid status on Data Center Default. Setting Data Center status to Non Responsive (On host rack, Error: Cannot access master storage domain). >2014-07-31 20:30:56,925 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-60) [1cb6c42c] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2014-07-31 20:30:56,928 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-60) [1cb6c42c] hostFromVds::selectedVds - rack, spmStatus returned null! >2014-07-31 20:30:56,929 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-60) [1cb6c42c] IrsBroker::Failed::GetStoragePoolInfoVDS >2014-07-31 20:30:56,930 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-60) [1cb6c42c] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2014-07-31 20:30:56,969 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-47) [756dc666] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage >2014-07-31 20:30:56,990 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-47) [756dc666] START, ResetIrsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, ignoreStopFailed = true), log id: 2de22ba1 >2014-07-31 20:30:56,994 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-47) [756dc666] START, SpmStopVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9), log id: 907165d >2014-07-31 20:30:57,001 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-47) [756dc666] SpmStopVDSCommand::Stopping SPM on vds rack, pool id 00000002-0002-0002-0002-0000000001d9 >2014-07-31 20:32:56,897 WARN [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-47) [756dc666] could not stop spm of pool 00000002-0002-0002-0002-0000000001d9 on vds 17f7fe56-5f32-40ef-b098-611b84484e0b - reason: org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcRunTimeException: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException >2014-07-31 20:32:56,906 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-5) Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: VDSNetworkException: org.apache.commons.httpclient.NoHttpResponseException: The server 10.34.62.208 failed to respond >2014-07-31 20:32:56,919 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-47) [756dc666] FINISH, SpmStopVDSCommand, log id: 907165d >2014-07-31 20:32:56,929 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-5) hostFromVds::selectedVds - rack, spmStatus returned null! >2014-07-31 20:32:56,931 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-47) [756dc666] FINISH, ResetIrsVDSCommand, log id: 2de22ba1 >2014-07-31 20:32:56,932 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ListVDSCommand] (DefaultQuartzScheduler_Worker-34) [5484a050] Command ListVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-07-31 20:32:56,942 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-34) [5484a050] Host rack is not responding. It will stay in Connecting state for a grace period of $80 seconds and after that an attempt to fence the host will be issued. >2014-07-31 20:32:56,947 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-34) [5484a050] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:406) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.fetchRunningVms(VdsUpdateRunTimeInfo.java:971) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVmStats(VdsUpdateRunTimeInfo.java:920) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:638) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:235) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) [:1.7.0_55] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_55] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_55] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_55] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_55] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_55] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_55] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_55] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy82.list(Unknown Source) > at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source) [:1.7.0_55] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_55] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_55] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_55] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_55] > >2014-07-31 20:32:56,961 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-47) [756dc666] Correlation ID: 756dc666, Job ID: 3d837794-9b23-4767-b9ee-fd56bc9befe3, Call Stack: null, Custom Event ID: -1, Message: Failed to Reconstruct Master Domain for Data Center Default. >2014-07-31 20:32:57,023 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-35) [1cb6c42c] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue >2014-07-31 20:32:57,027 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-26) starting processDomainRecovery for domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 >2014-07-31 20:32:57,032 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-34) [5484a050] Failed to refresh VDS , vds = 17f7fe56-5f32-40ef-b098-611b84484e0b : rack, VDS Network Error, continuing. >java.net.ConnectException: Connection refused >2014-07-31 20:32:57,095 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-26) Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 was reported by all hosts in status UP as problematic. Not moving the domain to NonOperational because it is being reconstructed now. >2014-07-31 20:32:57,113 WARN [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-26) [52fe06fe] CanDoAction of action ReconstructMasterDomain failed. Reasons:VAR__ACTION__RECONSTRUCT_MASTER,VAR__TYPE__STORAGE__DOMAIN,ACTION_TYPE_FAILED_NO_VDS_IN_POOL >2014-07-31 20:32:57,120 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-26) [52fe06fe] Removing vds [] from the domain in maintenance cache >2014-07-31 20:32:57,122 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-26) [52fe06fe] Removing host(s) [] from hosts unseen domain report cache >2014-07-31 20:32:57,124 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-35) [1cb6c42c] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue >2014-07-31 20:32:57,128 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-18) starting processDomainRecovery for domain 44559ec0-6bef-491c-bbd6-e67278f37987:export >2014-07-31 20:32:57,134 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-18) Domain 44559ec0-6bef-491c-bbd6-e67278f37987:export was reported by all hosts in status UP as problematic. Moving the domain to NonOperational. >2014-07-31 20:32:57,151 INFO [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-18) [65949611] Lock Acquired to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE >, sharedLocks= ] >2014-07-31 20:32:57,161 INFO [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-18) [65949611] Running command: DeactivateStorageDomainCommand internal: true. Entities affected : ID: 44559ec0-6bef-491c-bbd6-e67278f37987 Type: Storage >2014-07-31 20:32:57,169 INFO [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-18) [65949611] Lock freed to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE >, sharedLocks= ] >2014-07-31 20:32:57,180 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-18) [65949611] Correlation ID: 65949611, Job ID: c68098ca-bbe7-4bc9-8c1a-aed4a93b4c36, Call Stack: null, Custom Event ID: -1, Message: Storage Domain export (Data Center Default) was deactivated by system because it's not visible by any of the hosts. >2014-07-31 20:32:57,185 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-18) [65949611] Removing vds [] from the domain in maintenance cache >2014-07-31 20:32:57,186 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-18) [65949611] Removing host(s) [] from hosts unseen domain report cache >2014-07-31 20:33:00,301 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-23) [11b7484a] START, GetHardwareInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]), log id: 20e31637 >2014-07-31 20:33:00,353 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-23) [11b7484a] FINISH, GetHardwareInfoVDSCommand, log id: 20e31637 >2014-07-31 20:33:00,377 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-23) [3da73ce0] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS >2014-07-31 20:33:00,426 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-23) [7af5412a] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-07-31 20:33:00,435 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-23) [732318f1] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-07-31 20:33:00,446 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-23) [732318f1] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, storageType = NFS, connectionList = [{ id: 05047c9a-152e-4ef2-b118-4a1753c1eae0, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35export, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 7a212569 >2014-07-31 20:33:00,507 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-23) [732318f1] FINISH, ConnectStorageServerVDSCommand, return: {5083f370-af09-457b-9b73-3440e8b65a07=0, 05047c9a-152e-4ef2-b118-4a1753c1eae0=0}, log id: 7a212569 >2014-07-31 20:33:00,508 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-23) [732318f1] Host rack storage connection was succeeded >2014-07-31 20:33:00,516 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-1) START, ConnectStoragePoolVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, masterVersion = 1), log id: 17842b92 >2014-07-31 20:33:06,948 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-39) [4a2f7e33] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Inactive to Unknown, reason : no reporting hosts >2014-07-31 20:33:06,952 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-39) [4a2f7e33] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Inactive to Unknown, reason : no reporting hosts >2014-07-31 20:34:00,945 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-1) FINISH, ConnectStoragePoolVDSCommand, log id: 17842b92 >2014-07-31 20:34:00,958 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-23) [732318f1] START, SetMOMPolicyParametersVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 6ec70026 >2014-07-31 20:34:01,051 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-23) [732318f1] FINISH, SetMOMPolicyParametersVDSCommand, log id: 6ec70026 >2014-07-31 20:34:01,059 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-23) [732318f1] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host rack was set to Up. >2014-07-31 20:34:01,066 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-23) [732318f1] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host rack. >2014-07-31 20:34:01,081 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-23) [797bf730] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS >2014-07-31 20:34:07,071 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-89) [1b83a377] hostFromVds::selectedVds - rack, spmStatus Free, storage pool Default >2014-07-31 20:34:07,077 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-89) [1b83a377] starting spm on vds rack, storage pool Default, prevId -1, LVER -1 >2014-07-31 20:34:07,080 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-89) [1b83a377] START, SpmStartVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, prevId=-1, prevLVER=-1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 40095834 >2014-07-31 20:34:07,128 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-89) [1b83a377] spmStart polling started: taskId = cb2789cd-5a2d-402a-af18-fdd99a3ee7bf >2014-07-31 20:34:08,135 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-89) [1b83a377] spmStart polling ended: taskId = cb2789cd-5a2d-402a-af18-fdd99a3ee7bf task status = finished >2014-07-31 20:34:08,193 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-89) [1b83a377] spmStart polling ended, spm status: SPM >2014-07-31 20:34:08,195 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-89) [1b83a377] START, HSMClearTaskVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, taskId=cb2789cd-5a2d-402a-af18-fdd99a3ee7bf), log id: 327a561b >2014-07-31 20:34:08,241 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-89) [1b83a377] FINISH, HSMClearTaskVDSCommand, log id: 327a561b >2014-07-31 20:34:08,241 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-89) [1b83a377] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@1a59393c, log id: 40095834 >2014-07-31 20:34:08,245 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-89) [1b83a377] Initialize Irs proxy from vds: 10.34.62.208 >2014-07-31 20:34:08,253 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-89) [1b83a377] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208). >2014-07-31 20:34:08,254 WARN [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-8-thread-36) Executing a command: java.util.concurrent.FutureTask , but note that there are 1 tasks in the queue. >2014-07-31 20:34:08,264 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-36) [1b83a377] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 6120e90a >2014-07-31 20:34:08,296 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-36) [1b83a377] -- executeIrsBrokerCommand: Attempting on storage pool 00000002-0002-0002-0002-0000000001d9 >2014-07-31 20:34:08,298 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-36) [1b83a377] START, HSMGetAllTasksInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 5b4a57ff >2014-07-31 20:34:08,303 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-36) [1b83a377] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 5b4a57ff >2014-07-31 20:34:08,304 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-36) [1b83a377] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 6120e90a >2014-07-31 20:34:08,305 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (org.ovirt.thread.pool-8-thread-36) [1b83a377] Discovered no tasks on Storage Pool Default >2014-07-31 21:23:55,353 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-27) Principal admin::internal synchronized >2014-07-31 21:24:22,724 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-83) Attempting to update VMs/Templates Ovf. >2014-07-31 22:23:55,362 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-18) Principal admin::internal synchronized >2014-07-31 22:24:22,741 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-30) [dd7f8b1] Attempting to update VMs/Templates Ovf. >2014-07-31 23:23:55,369 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-27) Principal admin::internal synchronized >2014-07-31 23:24:22,758 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-28) [8ef1638] Attempting to update VMs/Templates Ovf. >2014-07-31 23:33:47,159 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-14) domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 in problem. vds: rack >2014-07-31 23:34:02,279 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-3) domain 44559ec0-6bef-491c-bbd6-e67278f37987:export in problem. vds: rack >2014-07-31 23:35:24,483 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-79) Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2014-07-31 23:35:24,487 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-79) [37510499] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-07-31 23:35:24,491 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-79) [37510499] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Active to Unknown, reason : null >2014-07-31 23:35:24,493 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-79) [37510499] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Active to Unknown, reason : null >2014-07-31 23:35:24,507 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-79) [37510499] Correlation ID: 37510499, Call Stack: null, Custom Event ID: -1, Message: Invalid status on Data Center Default. Setting Data Center status to Non Responsive (On host rack, Error: Cannot access master storage domain). >2014-07-31 23:37:24,536 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-79) [37510499] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2014-07-31 23:37:24,538 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-79) [37510499] hostFromVds::selectedVds - rack, spmStatus returned null! >2014-07-31 23:37:24,540 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-79) [37510499] IrsBroker::Failed::GetStoragePoolInfoVDS >2014-07-31 23:37:24,540 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-79) [37510499] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2014-07-31 23:37:24,590 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-11) [28c5ee0f] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage >2014-07-31 23:37:24,608 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-11) [28c5ee0f] START, ResetIrsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, ignoreStopFailed = true), log id: 24b0bb1d >2014-07-31 23:37:24,612 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-11) [28c5ee0f] START, SpmStopVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9), log id: 3c1908f0 >2014-07-31 23:37:24,618 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-11) [28c5ee0f] SpmStopVDSCommand::Stopping SPM on vds rack, pool id 00000002-0002-0002-0002-0000000001d9 >2014-07-31 23:39:24,509 WARN [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-11) [28c5ee0f] could not stop spm of pool 00000002-0002-0002-0002-0000000001d9 on vds 17f7fe56-5f32-40ef-b098-611b84484e0b - reason: org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcRunTimeException: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException >2014-07-31 23:39:24,512 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-11) [28c5ee0f] FINISH, SpmStopVDSCommand, log id: 3c1908f0 >2014-07-31 23:39:24,511 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ListVDSCommand] (DefaultQuartzScheduler_Worker-59) [1f7257b7] Command ListVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-07-31 23:39:24,509 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-3) Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: VDSNetworkException: org.apache.commons.httpclient.NoHttpResponseException: The server 10.34.62.208 failed to respond >2014-07-31 23:39:24,517 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-3) hostFromVds::selectedVds - rack, spmStatus returned null! >2014-07-31 23:39:24,515 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-59) [1f7257b7] Host rack is not responding. It will stay in Connecting state for a grace period of $80 seconds and after that an attempt to fence the host will be issued. >2014-07-31 23:39:24,519 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-11) [28c5ee0f] FINISH, ResetIrsVDSCommand, log id: 24b0bb1d >2014-07-31 23:39:24,520 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-59) [1f7257b7] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:406) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.fetchRunningVms(VdsUpdateRunTimeInfo.java:971) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVmStats(VdsUpdateRunTimeInfo.java:920) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:638) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:235) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) [:1.7.0_55] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_55] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_55] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_55] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_55] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_55] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_55] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_55] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy82.list(Unknown Source) > at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source) [:1.7.0_55] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_55] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_55] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_55] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_55] > >2014-07-31 23:39:24,540 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-11) [28c5ee0f] Correlation ID: 28c5ee0f, Job ID: 0628d3f4-ca61-4c68-a7c1-2cd663589ca1, Call Stack: null, Custom Event ID: -1, Message: Failed to Reconstruct Master Domain for Data Center Default. >2014-07-31 23:39:24,559 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-59) [1f7257b7] Failed to refresh VDS , vds = 17f7fe56-5f32-40ef-b098-611b84484e0b : rack, VDS Network Error, continuing. >java.net.ConnectException: Connection refused >2014-07-31 23:39:24,559 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-5) [37510499] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue >2014-07-31 23:39:24,562 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-46) starting processDomainRecovery for domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 >2014-07-31 23:39:24,617 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-46) Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 was reported by all hosts in status UP as problematic. Not moving the domain to NonOperational because it is being reconstructed now. >2014-07-31 23:39:24,636 WARN [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-46) [7235e194] CanDoAction of action ReconstructMasterDomain failed. Reasons:VAR__ACTION__RECONSTRUCT_MASTER,VAR__TYPE__STORAGE__DOMAIN,ACTION_TYPE_FAILED_NO_VDS_IN_POOL >2014-07-31 23:39:24,644 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-46) [7235e194] Removing vds [] from the domain in maintenance cache >2014-07-31 23:39:24,644 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-46) [7235e194] Removing host(s) [] from hosts unseen domain report cache >2014-07-31 23:39:24,645 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-5) [37510499] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue >2014-07-31 23:39:24,647 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-23) starting processDomainRecovery for domain 44559ec0-6bef-491c-bbd6-e67278f37987:export >2014-07-31 23:39:24,652 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-23) Domain 44559ec0-6bef-491c-bbd6-e67278f37987:export was reported by all hosts in status UP as problematic. Moving the domain to NonOperational. >2014-07-31 23:39:24,667 INFO [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-23) [323a2598] Lock Acquired to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE >, sharedLocks= ] >2014-07-31 23:39:24,673 INFO [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-23) [323a2598] Running command: DeactivateStorageDomainCommand internal: true. Entities affected : ID: 44559ec0-6bef-491c-bbd6-e67278f37987 Type: Storage >2014-07-31 23:39:24,679 INFO [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-23) [323a2598] Lock freed to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE >, sharedLocks= ] >2014-07-31 23:39:24,689 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-23) [323a2598] Correlation ID: 323a2598, Job ID: e7707ab5-d1d9-4555-8581-8f995aead2dc, Call Stack: null, Custom Event ID: -1, Message: Storage Domain export (Data Center Default) was deactivated by system because it's not visible by any of the hosts. >2014-07-31 23:39:24,698 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-23) [323a2598] Removing vds [] from the domain in maintenance cache >2014-07-31 23:39:24,699 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-23) [323a2598] Removing host(s) [] from hosts unseen domain report cache >2014-07-31 23:39:27,822 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-18) START, GetHardwareInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]), log id: 651073b7 >2014-07-31 23:39:27,878 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-18) FINISH, GetHardwareInfoVDSCommand, log id: 651073b7 >2014-07-31 23:39:27,904 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-18) [676113c7] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS >2014-07-31 23:39:27,954 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-18) [2bda1b13] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-07-31 23:39:27,963 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-18) [2985a5a2] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-07-31 23:39:27,975 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-18) [2985a5a2] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, storageType = NFS, connectionList = [{ id: 05047c9a-152e-4ef2-b118-4a1753c1eae0, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35export, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 76460e8f >2014-07-31 23:39:28,033 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-18) [2985a5a2] FINISH, ConnectStorageServerVDSCommand, return: {5083f370-af09-457b-9b73-3440e8b65a07=0, 05047c9a-152e-4ef2-b118-4a1753c1eae0=0}, log id: 76460e8f >2014-07-31 23:39:28,034 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-18) [2985a5a2] Host rack storage connection was succeeded >2014-07-31 23:39:28,042 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-41) START, ConnectStoragePoolVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, masterVersion = 1), log id: 2eb80d8f >2014-07-31 23:39:34,522 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-7) [42c2440e] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Inactive to Unknown, reason : no reporting hosts >2014-07-31 23:39:34,527 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-7) [42c2440e] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Inactive to Unknown, reason : no reporting hosts >2014-07-31 23:40:28,439 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-41) FINISH, ConnectStoragePoolVDSCommand, log id: 2eb80d8f >2014-07-31 23:40:28,452 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-18) [2985a5a2] START, SetMOMPolicyParametersVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 59540fe9 >2014-07-31 23:40:28,547 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-18) [2985a5a2] FINISH, SetMOMPolicyParametersVDSCommand, log id: 59540fe9 >2014-07-31 23:40:28,554 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-18) [2985a5a2] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host rack was set to Up. >2014-07-31 23:40:28,560 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-18) [2985a5a2] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host rack. >2014-07-31 23:40:28,576 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-18) [5497a121] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS >2014-07-31 23:40:34,675 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-51) [4760e42] hostFromVds::selectedVds - rack, spmStatus Free, storage pool Default >2014-07-31 23:40:34,681 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-51) [4760e42] starting spm on vds rack, storage pool Default, prevId -1, LVER -1 >2014-07-31 23:40:34,683 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-51) [4760e42] START, SpmStartVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, prevId=-1, prevLVER=-1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 5b3b1dcb >2014-07-31 23:40:34,732 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-51) [4760e42] spmStart polling started: taskId = 169337fe-d0af-4239-8f2c-cfd8e87ae4da >2014-07-31 23:40:35,738 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-51) [4760e42] spmStart polling ended: taskId = 169337fe-d0af-4239-8f2c-cfd8e87ae4da task status = finished >2014-07-31 23:40:35,794 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-51) [4760e42] spmStart polling ended, spm status: SPM >2014-07-31 23:40:35,796 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-51) [4760e42] START, HSMClearTaskVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, taskId=169337fe-d0af-4239-8f2c-cfd8e87ae4da), log id: 19538009 >2014-07-31 23:40:35,842 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-51) [4760e42] FINISH, HSMClearTaskVDSCommand, log id: 19538009 >2014-07-31 23:40:35,843 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-51) [4760e42] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@2dae15f9, log id: 5b3b1dcb >2014-07-31 23:40:35,846 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-51) [4760e42] Initialize Irs proxy from vds: 10.34.62.208 >2014-07-31 23:40:35,855 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-51) [4760e42] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208). >2014-07-31 23:40:35,863 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-29) [4760e42] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 6d97f318 >2014-07-31 23:40:35,901 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-29) [4760e42] -- executeIrsBrokerCommand: Attempting on storage pool 00000002-0002-0002-0002-0000000001d9 >2014-07-31 23:40:35,903 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-29) [4760e42] START, HSMGetAllTasksInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 16fa17c3 >2014-07-31 23:40:35,908 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-29) [4760e42] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 16fa17c3 >2014-07-31 23:40:35,909 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-29) [4760e42] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 6d97f318 >2014-07-31 23:40:35,910 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (org.ovirt.thread.pool-8-thread-29) [4760e42] Discovered no tasks on Storage Pool Default >2014-08-01 00:23:55,376 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-18) [5497a121] Principal admin::internal synchronized >2014-08-01 00:24:22,794 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-14) Attempting to update VMs/Templates Ovf. >2014-08-01 01:23:55,384 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-6) [7f1e369e] Principal admin::internal synchronized >2014-08-01 01:24:22,827 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-10) Attempting to update VMs/Templates Ovf. >2014-08-01 02:23:55,390 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-71) [6cd4f56d] Principal admin::internal synchronized >2014-08-01 02:24:22,842 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-52) Attempting to update VMs/Templates Ovf. >2014-08-01 02:40:20,402 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-16) domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 in problem. vds: rack >2014-08-01 02:40:20,405 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-16) domain 44559ec0-6bef-491c-bbd6-e67278f37987:export in problem. vds: rack >2014-08-01 02:41:50,243 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-77) [79b9f7da] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2014-08-01 02:41:50,248 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-77) [2e3238db] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-08-01 02:41:50,250 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-77) [2e3238db] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Active to Unknown, reason : null >2014-08-01 02:41:50,257 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-77) [2e3238db] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Active to Unknown, reason : null >2014-08-01 02:41:50,270 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-77) [2e3238db] Correlation ID: 2e3238db, Call Stack: null, Custom Event ID: -1, Message: Invalid status on Data Center Default. Setting Data Center status to Non Responsive (On host rack, Error: Cannot access master storage domain). >2014-08-01 02:43:50,295 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-77) [2e3238db] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2014-08-01 02:43:50,298 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-77) [2e3238db] hostFromVds::selectedVds - rack, spmStatus returned null! >2014-08-01 02:43:50,299 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-77) [2e3238db] IrsBroker::Failed::GetStoragePoolInfoVDS >2014-08-01 02:43:50,300 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-77) [2e3238db] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2014-08-01 02:43:50,354 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-10) [d926c49] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage >2014-08-01 02:43:50,377 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-10) [d926c49] START, ResetIrsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, ignoreStopFailed = true), log id: 36ad7aa7 >2014-08-01 02:43:50,381 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-10) [d926c49] START, SpmStopVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9), log id: 415ce57b >2014-08-01 02:43:50,387 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-10) [d926c49] SpmStopVDSCommand::Stopping SPM on vds rack, pool id 00000002-0002-0002-0002-0000000001d9 >2014-08-01 02:45:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-24) Autorecovering 1 storage domains >2014-08-01 02:45:00,005 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-24) Autorecovering storage domains id: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 >2014-08-01 02:45:00,007 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-24) [41bfb6e1] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage >2014-08-01 02:45:00,009 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-24) [41bfb6e1] ConnectDomainToStorage. Before Connect all hosts to pool. Time:8/1/14 2:45 AM >2014-08-01 02:45:00,018 INFO [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-8-thread-33) [40136750] Running command: ConnectStorageToVdsCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2014-08-01 02:45:00,022 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-33) [40136750] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 1127d8cd >2014-08-01 02:45:50,270 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-83) Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2014-08-01 02:45:50,272 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-83) hostFromVds::selectedVds - rack, spmStatus returned null! >2014-08-01 02:45:50,273 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-83) IrsBroker::Failed::GetStoragePoolInfoVDS >2014-08-01 02:45:50,274 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-83) Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2014-08-01 02:46:50,273 WARN [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-10) [d926c49] could not stop spm of pool 00000002-0002-0002-0002-0000000001d9 on vds 17f7fe56-5f32-40ef-b098-611b84484e0b - reason: org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcRunTimeException: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException >2014-08-01 02:46:50,283 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-4) Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-08-01 02:46:50,281 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ListVDSCommand] (DefaultQuartzScheduler_Worker-5) Command ListVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-08-01 02:46:50,278 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-33) [40136750] Command ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]) execution failed. Exception: VDSNetworkException: org.apache.commons.httpclient.NoHttpResponseException: The server 10.34.62.208 failed to respond >2014-08-01 02:46:50,290 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-5) Host rack is not responding. It will stay in Connecting state for a grace period of $80 seconds and after that an attempt to fence the host will be issued. >2014-08-01 02:46:50,286 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-4) hostFromVds::selectedVds - rack, spmStatus returned null! >2014-08-01 02:46:50,284 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-10) [d926c49] FINISH, SpmStopVDSCommand, log id: 415ce57b >2014-08-01 02:46:50,293 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-33) [40136750] FINISH, ConnectStorageServerVDSCommand, log id: 1127d8cd >2014-08-01 02:46:50,303 ERROR [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-8-thread-33) [40136750] Command org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: org.apache.commons.httpclient.NoHttpResponseException: The server 10.34.62.208 failed to respond (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-08-01 02:46:50,296 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-5) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:406) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.fetchRunningVms(VdsUpdateRunTimeInfo.java:971) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVmStats(VdsUpdateRunTimeInfo.java:920) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:638) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:235) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) [:1.7.0_55] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_55] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_55] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_55] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_55] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_55] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_55] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_55] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy82.list(Unknown Source) > at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source) [:1.7.0_55] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_55] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_55] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_55] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_55] > >2014-08-01 02:46:50,317 ERROR [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-8-thread-33) [40136750] Transaction rolled-back for command: org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand. >2014-08-01 02:46:50,308 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-10) [d926c49] FINISH, ResetIrsVDSCommand, log id: 36ad7aa7 >2014-08-01 02:46:50,339 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-24) [41bfb6e1] ConnectDomainToStorage. After Connect all hosts to pool. Time:8/1/14 2:46 AM >2014-08-01 02:46:50,346 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-5) Failed to refresh VDS , vds = 17f7fe56-5f32-40ef-b098-611b84484e0b : rack, VDS Network Error, continuing. >java.net.ConnectException: Connection refused >2014-08-01 02:46:50,352 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-10) [d926c49] Correlation ID: d926c49, Job ID: a6e60d1e-c692-417d-9a01-bb212d2eea4e, Call Stack: null, Custom Event ID: -1, Message: Failed to Reconstruct Master Domain for Data Center Default. >2014-08-01 02:46:50,360 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-48) [2e3238db] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue >2014-08-01 02:46:50,362 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-6) starting processDomainRecovery for domain 44559ec0-6bef-491c-bbd6-e67278f37987:export >2014-08-01 02:46:50,432 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-6) Domain 44559ec0-6bef-491c-bbd6-e67278f37987:export was reported by all hosts in status UP as problematic. Moving the domain to NonOperational. >2014-08-01 02:46:50,455 INFO [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-6) [5e1e7b65] Lock Acquired to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE >, sharedLocks= ] >2014-08-01 02:46:50,462 INFO [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-6) [5e1e7b65] Running command: DeactivateStorageDomainCommand internal: true. Entities affected : ID: 44559ec0-6bef-491c-bbd6-e67278f37987 Type: Storage >2014-08-01 02:46:50,469 INFO [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-6) [5e1e7b65] Lock freed to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE >, sharedLocks= ] >2014-08-01 02:46:50,479 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-6) [5e1e7b65] Correlation ID: 5e1e7b65, Job ID: bb9449be-55e4-4cdc-a79d-5bfecfab29a5, Call Stack: null, Custom Event ID: -1, Message: Storage Domain export (Data Center Default) was deactivated by system because it's not visible by any of the hosts. >2014-08-01 02:46:50,487 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-6) [5e1e7b65] Removing vds [] from the domain in maintenance cache >2014-08-01 02:46:50,488 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-6) [5e1e7b65] Removing host(s) [] from hosts unseen domain report cache >2014-08-01 02:46:50,490 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-12) starting processDomainRecovery for domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 >2014-08-01 02:46:50,495 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-12) Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 was reported by all hosts in status UP as problematic. Not moving the domain to NonOperational because it is being reconstructed now. >2014-08-01 02:46:50,518 WARN [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-12) [312d0f0] CanDoAction of action ReconstructMasterDomain failed. Reasons:VAR__ACTION__RECONSTRUCT_MASTER,VAR__TYPE__STORAGE__DOMAIN,ACTION_TYPE_FAILED_NO_VDS_IN_POOL >2014-08-01 02:46:50,525 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-12) [312d0f0] Removing vds [] from the domain in maintenance cache >2014-08-01 02:46:50,526 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-12) [312d0f0] Removing host(s) [] from hosts unseen domain report cache >2014-08-01 02:46:50,527 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-48) [2e3238db] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue >2014-08-01 02:46:53,598 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-66) [653adc38] START, GetHardwareInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]), log id: 513f447e >2014-08-01 02:46:53,654 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-66) [653adc38] FINISH, GetHardwareInfoVDSCommand, log id: 513f447e >2014-08-01 02:46:53,677 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-66) [230b5395] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS >2014-08-01 02:46:53,689 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-66) [5a39e60a] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-08-01 02:46:53,697 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-66) [6a720f1c] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-08-01 02:46:53,709 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-66) [6a720f1c] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, storageType = NFS, connectionList = [{ id: 05047c9a-152e-4ef2-b118-4a1753c1eae0, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35export, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 70a08d8 >2014-08-01 02:46:53,769 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-66) [6a720f1c] FINISH, ConnectStorageServerVDSCommand, return: {5083f370-af09-457b-9b73-3440e8b65a07=0, 05047c9a-152e-4ef2-b118-4a1753c1eae0=0}, log id: 70a08d8 >2014-08-01 02:46:53,770 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-66) [6a720f1c] Host rack storage connection was succeeded >2014-08-01 02:46:53,778 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-18) START, ConnectStoragePoolVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, masterVersion = 1), log id: 38974dad >2014-08-01 02:47:00,302 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-63) Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Inactive to Unknown, reason : no reporting hosts >2014-08-01 02:47:00,306 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-63) Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Inactive to Unknown, reason : no reporting hosts >2014-08-01 02:47:54,159 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-18) FINISH, ConnectStoragePoolVDSCommand, log id: 38974dad >2014-08-01 02:47:54,172 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-66) [6a720f1c] START, SetMOMPolicyParametersVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 7aed3c4 >2014-08-01 02:47:54,266 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-66) [6a720f1c] FINISH, SetMOMPolicyParametersVDSCommand, log id: 7aed3c4 >2014-08-01 02:47:54,273 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-66) [6a720f1c] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host rack was set to Up. >2014-08-01 02:47:54,279 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-66) [6a720f1c] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host rack. >2014-08-01 02:47:54,301 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-66) [2809936b] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS >2014-08-01 02:48:00,427 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-36) [3e2ffa8] hostFromVds::selectedVds - rack, spmStatus Free, storage pool Default >2014-08-01 02:48:00,432 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-36) [3e2ffa8] starting spm on vds rack, storage pool Default, prevId -1, LVER -1 >2014-08-01 02:48:00,435 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-36) [3e2ffa8] START, SpmStartVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, prevId=-1, prevLVER=-1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 3e2778e9 >2014-08-01 02:48:00,483 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-36) [3e2ffa8] spmStart polling started: taskId = 21499035-0c4b-4878-9f26-1840bd46bda9 >2014-08-01 02:48:01,489 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-36) [3e2ffa8] spmStart polling ended: taskId = 21499035-0c4b-4878-9f26-1840bd46bda9 task status = finished >2014-08-01 02:48:01,546 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-36) [3e2ffa8] spmStart polling ended, spm status: SPM >2014-08-01 02:48:01,548 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-36) [3e2ffa8] START, HSMClearTaskVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, taskId=21499035-0c4b-4878-9f26-1840bd46bda9), log id: 59174e32 >2014-08-01 02:48:01,594 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-36) [3e2ffa8] FINISH, HSMClearTaskVDSCommand, log id: 59174e32 >2014-08-01 02:48:01,595 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-36) [3e2ffa8] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@7bb3bf0e, log id: 3e2778e9 >2014-08-01 02:48:01,599 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-36) [3e2ffa8] Initialize Irs proxy from vds: 10.34.62.208 >2014-08-01 02:48:01,606 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-36) [3e2ffa8] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208). >2014-08-01 02:48:01,608 WARN [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-8-thread-13) Executing a command: java.util.concurrent.FutureTask , but note that there are 1 tasks in the queue. >2014-08-01 02:48:01,618 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-13) [3e2ffa8] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 68b90328 >2014-08-01 02:48:01,647 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-13) [3e2ffa8] -- executeIrsBrokerCommand: Attempting on storage pool 00000002-0002-0002-0002-0000000001d9 >2014-08-01 02:48:01,649 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-13) [3e2ffa8] START, HSMGetAllTasksInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 25cb6a90 >2014-08-01 02:48:01,654 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-13) [3e2ffa8] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 25cb6a90 >2014-08-01 02:48:01,655 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-13) [3e2ffa8] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 68b90328 >2014-08-01 02:48:01,656 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (org.ovirt.thread.pool-8-thread-13) [3e2ffa8] Discovered no tasks on Storage Pool Default >2014-08-01 03:23:55,396 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-46) [56f1581c] Principal admin::internal synchronized >2014-08-01 03:24:22,857 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-99) Attempting to update VMs/Templates Ovf. >2014-08-01 03:35:35,000 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-84) [7ef2d911] Start deleteAgedOutAuditLogs >2014-08-01 03:35:35,005 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-84) [7ef2d911] Finished deleteAgedOutAuditLogs >2014-08-01 03:35:35,001 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-75) [52dee25c] Start deleteAgedOutCommandEntities >2014-08-01 03:35:35,012 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-75) [52dee25c] Finished deleteAgedOutCommandEntities >2014-08-01 03:55:15,578 WARN [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-8-thread-38) Executing a command: java.util.concurrent.FutureTask , but note that there are 1 tasks in the queue. >2014-08-01 04:23:55,401 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-82) [3abdaaa6] Principal admin::internal synchronized >2014-08-01 04:24:22,887 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-17) Attempting to update VMs/Templates Ovf. >2014-08-01 05:23:55,411 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-79) [37510499] Principal admin::internal synchronized >2014-08-01 05:24:22,906 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-97) [1b0a6877] Attempting to update VMs/Templates Ovf. >2014-08-01 05:47:37,182 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-39) domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 in problem. vds: rack >2014-08-01 05:47:37,184 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-39) domain 44559ec0-6bef-491c-bbd6-e67278f37987:export in problem. vds: rack >2014-08-01 05:49:16,283 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-42) [3eb4f572] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2014-08-01 05:49:16,288 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-42) [161c5527] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-08-01 05:49:16,292 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-42) [161c5527] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Active to Unknown, reason : null >2014-08-01 05:49:16,295 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-42) [161c5527] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Active to Unknown, reason : null >2014-08-01 05:49:16,309 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-42) [161c5527] Correlation ID: 161c5527, Call Stack: null, Custom Event ID: -1, Message: Invalid status on Data Center Default. Setting Data Center status to Non Responsive (On host rack, Error: Cannot access master storage domain). >2014-08-01 05:51:16,335 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-42) [161c5527] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2014-08-01 05:51:16,338 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-42) [161c5527] hostFromVds::selectedVds - rack, spmStatus returned null! >2014-08-01 05:51:16,338 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-42) [161c5527] IrsBroker::Failed::GetStoragePoolInfoVDS >2014-08-01 05:51:16,339 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-42) [161c5527] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2014-08-01 05:51:16,392 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-50) [665db16f] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage >2014-08-01 05:51:16,414 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-50) [665db16f] START, ResetIrsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, ignoreStopFailed = true), log id: 6b16e474 >2014-08-01 05:51:16,417 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-50) [665db16f] START, SpmStopVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9), log id: 2398c319 >2014-08-01 05:51:16,423 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-50) [665db16f] SpmStopVDSCommand::Stopping SPM on vds rack, pool id 00000002-0002-0002-0002-0000000001d9 >2014-08-01 05:53:16,306 WARN [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-50) [665db16f] could not stop spm of pool 00000002-0002-0002-0002-0000000001d9 on vds 17f7fe56-5f32-40ef-b098-611b84484e0b - reason: org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcRunTimeException: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException >2014-08-01 05:53:16,309 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-50) [665db16f] FINISH, SpmStopVDSCommand, log id: 2398c319 >2014-08-01 05:53:16,311 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-80) Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: VDSNetworkException: org.apache.commons.httpclient.NoHttpResponseException: The server 10.34.62.208 failed to respond >2014-08-01 05:53:16,313 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-80) hostFromVds::selectedVds - rack, spmStatus returned null! >2014-08-01 05:53:16,315 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-50) [665db16f] FINISH, ResetIrsVDSCommand, log id: 6b16e474 >2014-08-01 05:53:16,314 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ListVDSCommand] (DefaultQuartzScheduler_Worker-49) Command ListVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-08-01 05:53:16,318 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-49) Host rack is not responding. It will stay in Connecting state for a grace period of $80 seconds and after that an attempt to fence the host will be issued. >2014-08-01 05:53:16,320 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-49) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:406) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.fetchRunningVms(VdsUpdateRunTimeInfo.java:971) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVmStats(VdsUpdateRunTimeInfo.java:920) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:638) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:235) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) [:1.7.0_55] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_55] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_55] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_55] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_55] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_55] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_55] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_55] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy82.list(Unknown Source) > at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source) [:1.7.0_55] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_55] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_55] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_55] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_55] > >2014-08-01 05:53:16,332 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-50) [665db16f] Correlation ID: 665db16f, Job ID: cafbb138-6b38-420f-a478-117c34d80f15, Call Stack: null, Custom Event ID: -1, Message: Failed to Reconstruct Master Domain for Data Center Default. >2014-08-01 05:53:16,353 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-49) Failed to refresh VDS , vds = 17f7fe56-5f32-40ef-b098-611b84484e0b : rack, VDS Network Error, continuing. >java.net.ConnectException: Connection refused >2014-08-01 05:53:16,354 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-49) [161c5527] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue >2014-08-01 05:53:16,356 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-19) starting processDomainRecovery for domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 >2014-08-01 05:53:16,425 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-19) Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 was reported by all hosts in status UP as problematic. Not moving the domain to NonOperational because it is being reconstructed now. >2014-08-01 05:53:16,444 WARN [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-19) [6ba0b3a2] CanDoAction of action ReconstructMasterDomain failed. Reasons:VAR__ACTION__RECONSTRUCT_MASTER,VAR__TYPE__STORAGE__DOMAIN,ACTION_TYPE_FAILED_NO_VDS_IN_POOL >2014-08-01 05:53:16,452 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-19) [6ba0b3a2] Removing vds [] from the domain in maintenance cache >2014-08-01 05:53:16,453 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-19) [6ba0b3a2] Removing host(s) [] from hosts unseen domain report cache >2014-08-01 05:53:16,454 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-49) [161c5527] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue >2014-08-01 05:53:16,456 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-22) starting processDomainRecovery for domain 44559ec0-6bef-491c-bbd6-e67278f37987:export >2014-08-01 05:53:16,461 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-22) Domain 44559ec0-6bef-491c-bbd6-e67278f37987:export was reported by all hosts in status UP as problematic. Moving the domain to NonOperational. >2014-08-01 05:53:16,477 INFO [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-22) [3ddd3b82] Lock Acquired to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE >, sharedLocks= ] >2014-08-01 05:53:16,490 INFO [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-22) [3ddd3b82] Running command: DeactivateStorageDomainCommand internal: true. Entities affected : ID: 44559ec0-6bef-491c-bbd6-e67278f37987 Type: Storage >2014-08-01 05:53:16,496 INFO [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-22) [3ddd3b82] Lock freed to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE >, sharedLocks= ] >2014-08-01 05:53:16,515 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-22) [3ddd3b82] Correlation ID: 3ddd3b82, Job ID: 43414010-2dea-431c-9147-65f9fd005f0d, Call Stack: null, Custom Event ID: -1, Message: Storage Domain export (Data Center Default) was deactivated by system because it's not visible by any of the hosts. >2014-08-01 05:53:16,523 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-22) [3ddd3b82] Removing vds [] from the domain in maintenance cache >2014-08-01 05:53:16,524 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-22) [3ddd3b82] Removing host(s) [] from hosts unseen domain report cache >2014-08-01 05:53:19,622 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-20) [3adf532b] START, GetHardwareInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]), log id: 5857efb6 >2014-08-01 05:53:19,678 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-20) [3adf532b] FINISH, GetHardwareInfoVDSCommand, log id: 5857efb6 >2014-08-01 05:53:19,708 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-20) [216d7974] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS >2014-08-01 05:53:19,757 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-20) [c701568] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-08-01 05:53:19,766 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-20) [feae1d7] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-08-01 05:53:19,783 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-20) [feae1d7] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, storageType = NFS, connectionList = [{ id: 05047c9a-152e-4ef2-b118-4a1753c1eae0, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35export, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 96f1342 >2014-08-01 05:53:19,847 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-20) [feae1d7] FINISH, ConnectStorageServerVDSCommand, return: {5083f370-af09-457b-9b73-3440e8b65a07=0, 05047c9a-152e-4ef2-b118-4a1753c1eae0=0}, log id: 96f1342 >2014-08-01 05:53:19,849 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-20) [feae1d7] Host rack storage connection was succeeded >2014-08-01 05:53:19,860 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-4) START, ConnectStoragePoolVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, masterVersion = 1), log id: 78694d3f >2014-08-01 05:53:26,319 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-18) [5497a121] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Inactive to Unknown, reason : no reporting hosts >2014-08-01 05:53:26,322 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-18) [5497a121] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Inactive to Unknown, reason : no reporting hosts >2014-08-01 05:54:20,273 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-4) FINISH, ConnectStoragePoolVDSCommand, log id: 78694d3f >2014-08-01 05:54:20,292 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-20) [feae1d7] START, SetMOMPolicyParametersVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 403faecb >2014-08-01 05:54:20,387 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-20) [feae1d7] FINISH, SetMOMPolicyParametersVDSCommand, log id: 403faecb >2014-08-01 05:54:20,394 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-20) [feae1d7] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host rack was set to Up. >2014-08-01 05:54:20,400 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-20) [feae1d7] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host rack. >2014-08-01 05:54:20,420 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-20) [367a2664] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS >2014-08-01 05:54:26,437 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-67) [2f0280a3] hostFromVds::selectedVds - rack, spmStatus Free, storage pool Default >2014-08-01 05:54:26,448 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-67) [2f0280a3] starting spm on vds rack, storage pool Default, prevId -1, LVER -1 >2014-08-01 05:54:26,450 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-67) [2f0280a3] START, SpmStartVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, prevId=-1, prevLVER=-1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 4291af7a >2014-08-01 05:54:26,516 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-67) [2f0280a3] spmStart polling started: taskId = 10b3c802-bda7-4ea0-81d5-f467afdb3e90 >2014-08-01 05:54:27,531 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-67) [2f0280a3] spmStart polling ended: taskId = 10b3c802-bda7-4ea0-81d5-f467afdb3e90 task status = finished >2014-08-01 05:54:27,588 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-67) [2f0280a3] spmStart polling ended, spm status: SPM >2014-08-01 05:54:27,590 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-67) [2f0280a3] START, HSMClearTaskVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, taskId=10b3c802-bda7-4ea0-81d5-f467afdb3e90), log id: 6644cbea >2014-08-01 05:54:27,636 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-67) [2f0280a3] FINISH, HSMClearTaskVDSCommand, log id: 6644cbea >2014-08-01 05:54:27,637 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-67) [2f0280a3] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@60323fca, log id: 4291af7a >2014-08-01 05:54:27,641 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-67) [2f0280a3] Initialize Irs proxy from vds: 10.34.62.208 >2014-08-01 05:54:27,648 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-67) [2f0280a3] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208). >2014-08-01 05:54:27,649 WARN [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-8-thread-31) Executing a command: java.util.concurrent.FutureTask , but note that there are 1 tasks in the queue. >2014-08-01 05:54:27,667 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-31) [2f0280a3] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 1d279b1f >2014-08-01 05:54:27,699 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-31) [2f0280a3] -- executeIrsBrokerCommand: Attempting on storage pool 00000002-0002-0002-0002-0000000001d9 >2014-08-01 05:54:27,701 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-31) [2f0280a3] START, HSMGetAllTasksInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 615f9f81 >2014-08-01 05:54:27,706 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-31) [2f0280a3] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 615f9f81 >2014-08-01 05:54:27,707 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-31) [2f0280a3] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 1d279b1f >2014-08-01 05:54:27,708 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (org.ovirt.thread.pool-8-thread-31) [2f0280a3] Discovered no tasks on Storage Pool Default >2014-08-01 06:23:55,417 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-45) [51d0e706] Principal admin::internal synchronized >2014-08-01 06:24:22,936 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-87) Attempting to update VMs/Templates Ovf. >2014-08-01 07:23:55,423 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-30) [dd7f8b1] Principal admin::internal synchronized >2014-08-01 07:24:22,951 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-62) [2e742a17] Attempting to update VMs/Templates Ovf. >2014-08-01 08:23:55,432 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-47) Principal admin::internal synchronized >2014-08-01 08:24:22,981 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-36) [3e2ffa8] Attempting to update VMs/Templates Ovf. >2014-08-01 08:54:12,367 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-27) domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 in problem. vds: rack >2014-08-01 08:54:12,369 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-27) domain 44559ec0-6bef-491c-bbd6-e67278f37987:export in problem. vds: rack >2014-08-01 08:55:43,233 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-93) [341aec49] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2014-08-01 08:55:43,237 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-93) [156fe0ca] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-08-01 08:55:43,245 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-93) [156fe0ca] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Active to Unknown, reason : null >2014-08-01 08:55:43,249 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-93) [156fe0ca] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Active to Unknown, reason : null >2014-08-01 08:55:43,266 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-93) [156fe0ca] Correlation ID: 156fe0ca, Call Stack: null, Custom Event ID: -1, Message: Invalid status on Data Center Default. Setting Data Center status to Non Responsive (On host rack, Error: Cannot access master storage domain). >2014-08-01 08:57:43,292 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-93) [156fe0ca] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2014-08-01 08:57:43,295 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-93) [156fe0ca] hostFromVds::selectedVds - rack, spmStatus returned null! >2014-08-01 08:57:43,296 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-93) [156fe0ca] IrsBroker::Failed::GetStoragePoolInfoVDS >2014-08-01 08:57:43,298 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-93) [156fe0ca] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2014-08-01 08:57:43,333 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-3) [726a0f53] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage >2014-08-01 08:57:43,350 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-3) [726a0f53] START, ResetIrsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, ignoreStopFailed = true), log id: 66ddc035 >2014-08-01 08:57:43,354 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-3) [726a0f53] START, SpmStopVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9), log id: 5c0aff22 >2014-08-01 08:57:43,360 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-3) [726a0f53] SpmStopVDSCommand::Stopping SPM on vds rack, pool id 00000002-0002-0002-0002-0000000001d9 >2014-08-01 08:59:43,257 WARN [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-3) [726a0f53] could not stop spm of pool 00000002-0002-0002-0002-0000000001d9 on vds 17f7fe56-5f32-40ef-b098-611b84484e0b - reason: org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcRunTimeException: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException >2014-08-01 08:59:43,260 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-3) [726a0f53] FINISH, SpmStopVDSCommand, log id: 5c0aff22 >2014-08-01 08:59:43,259 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ListVDSCommand] (DefaultQuartzScheduler_Worker-27) Command ListVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-08-01 08:59:43,258 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-65) Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: VDSNetworkException: org.apache.commons.httpclient.NoHttpResponseException: The server 10.34.62.208 failed to respond >2014-08-01 08:59:43,266 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-65) hostFromVds::selectedVds - rack, spmStatus returned null! >2014-08-01 08:59:43,266 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-3) [726a0f53] FINISH, ResetIrsVDSCommand, log id: 66ddc035 >2014-08-01 08:59:43,265 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-27) Host rack is not responding. It will stay in Connecting state for a grace period of $80 seconds and after that an attempt to fence the host will be issued. >2014-08-01 08:59:43,270 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-27) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:406) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.fetchRunningVms(VdsUpdateRunTimeInfo.java:971) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVmStats(VdsUpdateRunTimeInfo.java:920) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:638) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:235) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) [:1.7.0_55] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_55] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_55] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_55] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_55] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_55] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_55] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_55] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy82.list(Unknown Source) > at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source) [:1.7.0_55] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_55] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_55] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_55] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_55] > >2014-08-01 08:59:43,282 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-3) [726a0f53] Correlation ID: 726a0f53, Job ID: 94ff1dbb-8864-493e-a5ee-83729ef01419, Call Stack: null, Custom Event ID: -1, Message: Failed to Reconstruct Master Domain for Data Center Default. >2014-08-01 08:59:43,302 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-27) Failed to refresh VDS , vds = 17f7fe56-5f32-40ef-b098-611b84484e0b : rack, VDS Network Error, continuing. >java.net.ConnectException: Connection refused >2014-08-01 08:59:43,304 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-35) [156fe0ca] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue >2014-08-01 08:59:43,306 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-27) starting processDomainRecovery for domain 44559ec0-6bef-491c-bbd6-e67278f37987:export >2014-08-01 08:59:43,373 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-27) Domain 44559ec0-6bef-491c-bbd6-e67278f37987:export was reported by all hosts in status UP as problematic. Moving the domain to NonOperational. >2014-08-01 08:59:43,389 INFO [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-27) [7dac8714] Lock Acquired to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE >, sharedLocks= ] >2014-08-01 08:59:43,397 INFO [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-27) [7dac8714] Running command: DeactivateStorageDomainCommand internal: true. Entities affected : ID: 44559ec0-6bef-491c-bbd6-e67278f37987 Type: Storage >2014-08-01 08:59:43,402 INFO [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-27) [7dac8714] Lock freed to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE >, sharedLocks= ] >2014-08-01 08:59:43,411 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-27) [7dac8714] Correlation ID: 7dac8714, Job ID: 44835dc1-8665-46e7-a0be-79bd9f034e2b, Call Stack: null, Custom Event ID: -1, Message: Storage Domain export (Data Center Default) was deactivated by system because it's not visible by any of the hosts. >2014-08-01 08:59:43,417 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-27) [7dac8714] Removing vds [] from the domain in maintenance cache >2014-08-01 08:59:43,417 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-27) [7dac8714] Removing host(s) [] from hosts unseen domain report cache >2014-08-01 08:59:43,419 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-28) starting processDomainRecovery for domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 >2014-08-01 08:59:43,425 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-28) Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 was reported by all hosts in status UP as problematic. Not moving the domain to NonOperational because it is being reconstructed now. >2014-08-01 08:59:43,443 WARN [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-28) [65e6b62e] CanDoAction of action ReconstructMasterDomain failed. Reasons:VAR__ACTION__RECONSTRUCT_MASTER,VAR__TYPE__STORAGE__DOMAIN,ACTION_TYPE_FAILED_NO_VDS_IN_POOL >2014-08-01 08:59:43,452 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-28) [65e6b62e] Removing vds [] from the domain in maintenance cache >2014-08-01 08:59:43,453 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-28) [65e6b62e] Removing host(s) [] from hosts unseen domain report cache >2014-08-01 08:59:43,453 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-35) [156fe0ca] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue >2014-08-01 08:59:46,562 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-67) [2f0280a3] START, GetHardwareInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]), log id: 3a9e57f0 >2014-08-01 08:59:46,621 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-67) [2f0280a3] FINISH, GetHardwareInfoVDSCommand, log id: 3a9e57f0 >2014-08-01 08:59:46,645 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-67) [39e62ae6] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS >2014-08-01 08:59:46,657 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-67) [25a43784] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-08-01 08:59:46,665 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-67) [13f100ee] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-08-01 08:59:46,677 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-67) [13f100ee] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, storageType = NFS, connectionList = [{ id: 05047c9a-152e-4ef2-b118-4a1753c1eae0, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35export, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 2f00b828 >2014-08-01 08:59:46,736 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-67) [13f100ee] FINISH, ConnectStorageServerVDSCommand, return: {5083f370-af09-457b-9b73-3440e8b65a07=0, 05047c9a-152e-4ef2-b118-4a1753c1eae0=0}, log id: 2f00b828 >2014-08-01 08:59:46,738 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-67) [13f100ee] Host rack storage connection was succeeded >2014-08-01 08:59:46,746 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-22) START, ConnectStoragePoolVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, masterVersion = 1), log id: 39b166f0 >2014-08-01 08:59:53,272 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-72) Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Inactive to Unknown, reason : no reporting hosts >2014-08-01 08:59:53,275 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-72) Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Inactive to Unknown, reason : no reporting hosts >2014-08-01 09:00:47,174 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-22) FINISH, ConnectStoragePoolVDSCommand, log id: 39b166f0 >2014-08-01 09:00:47,188 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-67) [13f100ee] START, SetMOMPolicyParametersVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: c8d4332 >2014-08-01 09:00:47,281 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-67) [13f100ee] FINISH, SetMOMPolicyParametersVDSCommand, log id: c8d4332 >2014-08-01 09:00:47,290 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-67) [13f100ee] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host rack was set to Up. >2014-08-01 09:00:47,296 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-67) [13f100ee] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host rack. >2014-08-01 09:00:47,321 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-67) [6199900c] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS >2014-08-01 09:00:53,422 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-81) hostFromVds::selectedVds - rack, spmStatus Free, storage pool Default >2014-08-01 09:00:53,428 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-81) starting spm on vds rack, storage pool Default, prevId -1, LVER -1 >2014-08-01 09:00:53,430 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-81) START, SpmStartVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, prevId=-1, prevLVER=-1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 544ea85a >2014-08-01 09:00:53,479 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-81) spmStart polling started: taskId = 58f02b46-9ba6-42d7-8910-03b6ca4bc88e >2014-08-01 09:00:54,486 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-81) spmStart polling ended: taskId = 58f02b46-9ba6-42d7-8910-03b6ca4bc88e task status = finished >2014-08-01 09:00:54,542 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-81) spmStart polling ended, spm status: SPM >2014-08-01 09:00:54,544 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-81) START, HSMClearTaskVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, taskId=58f02b46-9ba6-42d7-8910-03b6ca4bc88e), log id: 3eec0a34 >2014-08-01 09:00:54,589 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-81) FINISH, HSMClearTaskVDSCommand, log id: 3eec0a34 >2014-08-01 09:00:54,590 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-81) FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@7ed410e8, log id: 544ea85a >2014-08-01 09:00:54,595 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-81) Initialize Irs proxy from vds: 10.34.62.208 >2014-08-01 09:00:54,601 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-81) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208). >2014-08-01 09:00:54,611 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-38) START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 3f91f119 >2014-08-01 09:00:54,645 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-38) -- executeIrsBrokerCommand: Attempting on storage pool 00000002-0002-0002-0002-0000000001d9 >2014-08-01 09:00:54,647 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-38) START, HSMGetAllTasksInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 1c10698 >2014-08-01 09:00:54,653 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-38) FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 1c10698 >2014-08-01 09:00:54,654 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-38) FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 3f91f119 >2014-08-01 09:00:54,655 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (org.ovirt.thread.pool-8-thread-38) Discovered no tasks on Storage Pool Default >2014-08-01 09:23:55,439 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-1) Principal admin::internal synchronized >2014-08-01 09:24:22,996 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-23) [797bf730] Attempting to update VMs/Templates Ovf. >2014-08-01 10:23:55,446 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-9) Principal admin::internal synchronized >2014-08-01 10:24:23,016 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-48) Attempting to update VMs/Templates Ovf. >2014-08-01 11:23:55,452 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-100) [52a75f18] Principal admin::internal synchronized >2014-08-01 11:24:23,031 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-79) [37510499] Attempting to update VMs/Templates Ovf. >2014-08-01 12:00:44,899 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-36) domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 in problem. vds: rack >2014-08-01 12:01:00,020 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-18) domain 44559ec0-6bef-491c-bbd6-e67278f37987:export in problem. vds: rack >2014-08-01 12:02:15,458 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-45) [51d0e706] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2014-08-01 12:02:15,463 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-45) [18f9dc27] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-08-01 12:02:15,466 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-45) [18f9dc27] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Active to Unknown, reason : null >2014-08-01 12:02:15,469 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-45) [18f9dc27] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Active to Unknown, reason : null >2014-08-01 12:02:15,481 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-45) [18f9dc27] Correlation ID: 18f9dc27, Call Stack: null, Custom Event ID: -1, Message: Invalid status on Data Center Default. Setting Data Center status to Non Responsive (On host rack, Error: Cannot access master storage domain). >2014-08-01 12:04:15,513 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-45) [18f9dc27] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2014-08-01 12:04:15,516 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-45) [18f9dc27] hostFromVds::selectedVds - rack, spmStatus returned null! >2014-08-01 12:04:15,517 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-45) [18f9dc27] IrsBroker::Failed::GetStoragePoolInfoVDS >2014-08-01 12:04:15,517 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-45) [18f9dc27] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2014-08-01 12:04:15,590 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-25) [38a1f3c4] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage >2014-08-01 12:04:15,692 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-25) [38a1f3c4] START, ResetIrsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, ignoreStopFailed = true), log id: 2e2d7f1c >2014-08-01 12:04:15,695 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-25) [38a1f3c4] START, SpmStopVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9), log id: 16618ea >2014-08-01 12:04:15,701 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-25) [38a1f3c4] SpmStopVDSCommand::Stopping SPM on vds rack, pool id 00000002-0002-0002-0002-0000000001d9 >2014-08-01 12:05:00,014 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-15) Autorecovering 1 storage domains >2014-08-01 12:05:00,016 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-15) Autorecovering storage domains id: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 >2014-08-01 12:05:00,018 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-15) [39225a1e] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage >2014-08-01 12:05:00,021 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-15) [39225a1e] ConnectDomainToStorage. Before Connect all hosts to pool. Time:8/1/14 12:05 PM >2014-08-01 12:05:00,034 INFO [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-8-thread-2) [17f6246] Running command: ConnectStorageToVdsCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2014-08-01 12:05:00,039 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-2) [17f6246] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 3a2232aa >2014-08-01 12:06:15,487 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-81) Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: VDSNetworkException: org.apache.commons.httpclient.NoHttpResponseException: The server 10.34.62.208 failed to respond >2014-08-01 12:06:15,492 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ListVDSCommand] (DefaultQuartzScheduler_Worker-52) Command ListVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-08-01 12:06:15,495 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-52) Host rack is not responding. It will stay in Connecting state for a grace period of $80 seconds and after that an attempt to fence the host will be issued. >2014-08-01 12:06:15,488 WARN [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-25) [38a1f3c4] could not stop spm of pool 00000002-0002-0002-0002-0000000001d9 on vds 17f7fe56-5f32-40ef-b098-611b84484e0b - reason: org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcRunTimeException: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException >2014-08-01 12:06:15,499 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-25) [38a1f3c4] FINISH, SpmStopVDSCommand, log id: 16618ea >2014-08-01 12:06:15,492 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-2) [17f6246] Command ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-08-01 12:06:15,493 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-81) hostFromVds::selectedVds - rack, spmStatus returned null! >2014-08-01 12:06:15,505 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-2) [17f6246] FINISH, ConnectStorageServerVDSCommand, log id: 3a2232aa >2014-08-01 12:06:15,509 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-25) [38a1f3c4] FINISH, ResetIrsVDSCommand, log id: 2e2d7f1c >2014-08-01 12:06:15,497 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-52) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:406) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.fetchRunningVms(VdsUpdateRunTimeInfo.java:971) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVmStats(VdsUpdateRunTimeInfo.java:920) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:638) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:235) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) [:1.7.0_55] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_55] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_55] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_55] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_55] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_55] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_55] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_55] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy82.list(Unknown Source) > at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source) [:1.7.0_55] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_55] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_55] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_55] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_55] > >2014-08-01 12:06:15,518 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-25) [38a1f3c4] Correlation ID: 38a1f3c4, Job ID: 5d187e6b-dfcf-48bc-af17-d723a1138731, Call Stack: null, Custom Event ID: -1, Message: Failed to Reconstruct Master Domain for Data Center Default. >2014-08-01 12:06:15,510 ERROR [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-8-thread-2) [17f6246] Command org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-08-01 12:06:15,541 ERROR [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-8-thread-2) [17f6246] Transaction rolled-back for command: org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand. >2014-08-01 12:06:15,543 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-15) [39225a1e] ConnectDomainToStorage. After Connect all hosts to pool. Time:8/1/14 12:06 PM >2014-08-01 12:06:15,543 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-32) [18f9dc27] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue >2014-08-01 12:06:15,546 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-36) starting processDomainRecovery for domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 >2014-08-01 12:06:15,547 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-52) Failed to refresh VDS , vds = 17f7fe56-5f32-40ef-b098-611b84484e0b : rack, VDS Network Error, continuing. >java.net.ConnectException: Connection refused >2014-08-01 12:06:15,610 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-36) Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 was reported by all hosts in status UP as problematic. Not moving the domain to NonOperational because it is being reconstructed now. >2014-08-01 12:06:15,637 WARN [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-36) [a43af] CanDoAction of action ReconstructMasterDomain failed. Reasons:VAR__ACTION__RECONSTRUCT_MASTER,VAR__TYPE__STORAGE__DOMAIN,ACTION_TYPE_FAILED_NO_VDS_IN_POOL >2014-08-01 12:06:15,645 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-36) [a43af] Removing vds [] from the domain in maintenance cache >2014-08-01 12:06:15,646 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-36) [a43af] Removing host(s) [] from hosts unseen domain report cache >2014-08-01 12:06:15,647 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-32) [18f9dc27] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue >2014-08-01 12:06:15,649 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-36) starting processDomainRecovery for domain 44559ec0-6bef-491c-bbd6-e67278f37987:export >2014-08-01 12:06:15,654 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-36) Domain 44559ec0-6bef-491c-bbd6-e67278f37987:export was reported by all hosts in status UP as problematic. Moving the domain to NonOperational. >2014-08-01 12:06:15,673 INFO [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-36) [7951b282] Lock Acquired to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE >, sharedLocks= ] >2014-08-01 12:06:15,682 INFO [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-36) [7951b282] Running command: DeactivateStorageDomainCommand internal: true. Entities affected : ID: 44559ec0-6bef-491c-bbd6-e67278f37987 Type: Storage >2014-08-01 12:06:15,693 INFO [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-36) [7951b282] Lock freed to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE >, sharedLocks= ] >2014-08-01 12:06:15,710 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-36) [7951b282] Correlation ID: 7951b282, Job ID: 2be04eff-7eb2-448f-88d4-e7fc1c1b7628, Call Stack: null, Custom Event ID: -1, Message: Storage Domain export (Data Center Default) was deactivated by system because it's not visible by any of the hosts. >2014-08-01 12:06:15,718 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-36) [7951b282] Removing vds [] from the domain in maintenance cache >2014-08-01 12:06:15,720 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-36) [7951b282] Removing host(s) [] from hosts unseen domain report cache >2014-08-01 12:06:19,038 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-25) [8cb28d1] START, GetHardwareInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]), log id: 4191f0b4 >2014-08-01 12:06:19,093 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-25) [8cb28d1] FINISH, GetHardwareInfoVDSCommand, log id: 4191f0b4 >2014-08-01 12:06:19,122 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-25) [97e99d1] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS >2014-08-01 12:06:19,172 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-25) [5b66a69e] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-08-01 12:06:19,180 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-25) [7333b236] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-08-01 12:06:19,186 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-25) [7333b236] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, storageType = NFS, connectionList = [{ id: 05047c9a-152e-4ef2-b118-4a1753c1eae0, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35export, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 43da23a9 >2014-08-01 12:06:19,248 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-25) [7333b236] FINISH, ConnectStorageServerVDSCommand, return: {5083f370-af09-457b-9b73-3440e8b65a07=0, 05047c9a-152e-4ef2-b118-4a1753c1eae0=0}, log id: 43da23a9 >2014-08-01 12:06:19,249 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-25) [7333b236] Host rack storage connection was succeeded >2014-08-01 12:06:19,256 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-12) START, ConnectStoragePoolVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, masterVersion = 1), log id: 1c6f2a34 >2014-08-01 12:06:25,511 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-59) [1f7257b7] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Inactive to Unknown, reason : no reporting hosts >2014-08-01 12:06:25,515 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-59) [1f7257b7] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Inactive to Unknown, reason : no reporting hosts >2014-08-01 12:07:19,695 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-12) FINISH, ConnectStoragePoolVDSCommand, log id: 1c6f2a34 >2014-08-01 12:07:19,709 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-25) [7333b236] START, SetMOMPolicyParametersVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: a44339e >2014-08-01 12:07:19,801 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-25) [7333b236] FINISH, SetMOMPolicyParametersVDSCommand, log id: a44339e >2014-08-01 12:07:19,811 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-25) [7333b236] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host rack was set to Up. >2014-08-01 12:07:19,816 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-25) [7333b236] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host rack. >2014-08-01 12:07:19,834 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-25) [50daf5eb] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS >2014-08-01 12:07:25,635 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-47) hostFromVds::selectedVds - rack, spmStatus Free, storage pool Default >2014-08-01 12:07:25,640 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-47) starting spm on vds rack, storage pool Default, prevId -1, LVER -1 >2014-08-01 12:07:25,642 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-47) START, SpmStartVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, prevId=-1, prevLVER=-1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 4f1640e8 >2014-08-01 12:07:25,692 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-47) spmStart polling started: taskId = 5a8194b9-d70d-47fb-a7a7-6fd0dbd97f67 >2014-08-01 12:07:26,698 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-47) spmStart polling ended: taskId = 5a8194b9-d70d-47fb-a7a7-6fd0dbd97f67 task status = finished >2014-08-01 12:07:26,754 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-47) spmStart polling ended, spm status: SPM >2014-08-01 12:07:26,756 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-47) START, HSMClearTaskVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, taskId=5a8194b9-d70d-47fb-a7a7-6fd0dbd97f67), log id: 7d684184 >2014-08-01 12:07:26,802 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-47) FINISH, HSMClearTaskVDSCommand, log id: 7d684184 >2014-08-01 12:07:26,803 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-47) FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@6a058eaf, log id: 4f1640e8 >2014-08-01 12:07:26,806 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-47) Initialize Irs proxy from vds: 10.34.62.208 >2014-08-01 12:07:26,814 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-47) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208). >2014-08-01 12:07:26,816 WARN [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-8-thread-49) Executing a command: java.util.concurrent.FutureTask , but note that there are 1 tasks in the queue. >2014-08-01 12:07:26,829 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-49) START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 3aaec3e >2014-08-01 12:07:26,855 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-49) -- executeIrsBrokerCommand: Attempting on storage pool 00000002-0002-0002-0002-0000000001d9 >2014-08-01 12:07:26,857 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-49) START, HSMGetAllTasksInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 6c15f4d3 >2014-08-01 12:07:26,862 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-49) FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 6c15f4d3 >2014-08-01 12:07:26,863 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-49) FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 3aaec3e >2014-08-01 12:07:26,864 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (org.ovirt.thread.pool-8-thread-49) Discovered no tasks on Storage Pool Default >2014-08-01 12:23:55,460 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-23) [797bf730] Principal admin::internal synchronized >2014-08-01 12:24:23,044 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-61) [58bfe349] Attempting to update VMs/Templates Ovf. >2014-08-01 12:44:10,749 WARN [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-8-thread-43) Executing a command: java.util.concurrent.FutureTask , but note that there are 1 tasks in the queue. >2014-08-01 13:23:55,471 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-37) Principal admin::internal synchronized >2014-08-01 13:24:23,078 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-39) [4a2f7e33] Attempting to update VMs/Templates Ovf. >2014-08-01 14:23:55,478 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-50) [375fcc6] Principal admin::internal synchronized >2014-08-01 14:24:23,092 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-69) [52d379dc] Attempting to update VMs/Templates Ovf. >2014-08-01 15:07:05,529 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-22) domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 in problem. vds: rack >2014-08-01 15:07:20,660 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-42) domain 44559ec0-6bef-491c-bbd6-e67278f37987:export in problem. vds: rack >2014-08-01 15:08:41,970 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-35) Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2014-08-01 15:08:41,974 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-35) [78e7d47b] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-08-01 15:08:41,977 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-35) [78e7d47b] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Active to Unknown, reason : null >2014-08-01 15:08:41,979 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-35) [78e7d47b] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Active to Unknown, reason : null >2014-08-01 15:08:41,998 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-35) [78e7d47b] Correlation ID: 78e7d47b, Call Stack: null, Custom Event ID: -1, Message: Invalid status on Data Center Default. Setting Data Center status to Non Responsive (On host rack, Error: Cannot access master storage domain). >2014-08-01 15:10:42,024 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-35) [78e7d47b] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2014-08-01 15:10:42,026 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-35) [78e7d47b] hostFromVds::selectedVds - rack, spmStatus returned null! >2014-08-01 15:10:42,027 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-35) [78e7d47b] IrsBroker::Failed::GetStoragePoolInfoVDS >2014-08-01 15:10:42,028 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-35) [78e7d47b] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2014-08-01 15:10:42,057 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-44) [24a49849] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage >2014-08-01 15:10:42,074 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-44) [24a49849] START, ResetIrsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, ignoreStopFailed = true), log id: 4eb880d7 >2014-08-01 15:10:42,077 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-44) [24a49849] START, SpmStopVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9), log id: 26d2a09 >2014-08-01 15:10:42,083 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-44) [24a49849] SpmStopVDSCommand::Stopping SPM on vds rack, pool id 00000002-0002-0002-0002-0000000001d9 >2014-08-01 15:12:41,995 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-71) [6cd4f56d] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: VDSNetworkException: org.apache.commons.httpclient.NoHttpResponseException: The server 10.34.62.208 failed to respond >2014-08-01 15:12:41,997 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-71) [6cd4f56d] hostFromVds::selectedVds - rack, spmStatus returned null! >2014-08-01 15:12:41,998 WARN [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-44) [24a49849] could not stop spm of pool 00000002-0002-0002-0002-0000000001d9 on vds 17f7fe56-5f32-40ef-b098-611b84484e0b - reason: org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcRunTimeException: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException >2014-08-01 15:12:41,999 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetStatsVDSCommand] (DefaultQuartzScheduler_Worker-47) Command GetStatsVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-08-01 15:12:42,004 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-47) Host rack is not responding. It will stay in Connecting state for a grace period of $80 seconds and after that an attempt to fence the host will be issued. >2014-08-01 15:12:42,005 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-47) vds::refreshVdsStats Failed getVdsStats, vds = 17f7fe56-5f32-40ef-b098-611b84484e0b : rack, error = VDSNetworkException: java.net.ConnectException: Connection refused >2014-08-01 15:12:42,001 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-44) [24a49849] FINISH, SpmStopVDSCommand, log id: 26d2a09 >2014-08-01 15:12:42,006 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-47) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsStats(VdsUpdateRunTimeInfo.java:669) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:623) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:235) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) [:1.7.0_55] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_55] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_55] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_55] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_55] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_55] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_55] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_55] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy82.getVdsStats(Unknown Source) > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_55] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_55] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_55] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_55] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_55] > >2014-08-01 15:12:42,013 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-44) [24a49849] FINISH, ResetIrsVDSCommand, log id: 4eb880d7 >2014-08-01 15:12:42,035 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-47) Failed to refresh VDS , vds = 17f7fe56-5f32-40ef-b098-611b84484e0b : rack, VDS Network Error, continuing. >java.net.ConnectException: Connection refused >2014-08-01 15:12:42,042 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-44) [24a49849] Correlation ID: 24a49849, Job ID: 5e3f6925-8b9b-4940-86ee-f3972c1a690a, Call Stack: null, Custom Event ID: -1, Message: Failed to Reconstruct Master Domain for Data Center Default. >2014-08-01 15:12:42,047 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-28) [78e7d47b] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue >2014-08-01 15:12:42,049 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-40) starting processDomainRecovery for domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 >2014-08-01 15:12:42,117 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-40) Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 was reported by all hosts in status UP as problematic. Not moving the domain to NonOperational because it is being reconstructed now. >2014-08-01 15:12:42,137 WARN [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-40) [29a8744c] CanDoAction of action ReconstructMasterDomain failed. Reasons:VAR__ACTION__RECONSTRUCT_MASTER,VAR__TYPE__STORAGE__DOMAIN,ACTION_TYPE_FAILED_NO_VDS_IN_POOL >2014-08-01 15:12:42,144 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-40) [29a8744c] Removing vds [] from the domain in maintenance cache >2014-08-01 15:12:42,144 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-40) [29a8744c] Removing host(s) [] from hosts unseen domain report cache >2014-08-01 15:12:42,145 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-28) [78e7d47b] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue >2014-08-01 15:12:42,147 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-48) starting processDomainRecovery for domain 44559ec0-6bef-491c-bbd6-e67278f37987:export >2014-08-01 15:12:42,152 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-48) Domain 44559ec0-6bef-491c-bbd6-e67278f37987:export was reported by all hosts in status UP as problematic. Moving the domain to NonOperational. >2014-08-01 15:12:42,167 INFO [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-48) [3b8d5173] Lock Acquired to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE >, sharedLocks= ] >2014-08-01 15:12:42,174 INFO [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-48) [3b8d5173] Running command: DeactivateStorageDomainCommand internal: true. Entities affected : ID: 44559ec0-6bef-491c-bbd6-e67278f37987 Type: Storage >2014-08-01 15:12:42,179 INFO [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-48) [3b8d5173] Lock freed to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE >, sharedLocks= ] >2014-08-01 15:12:42,188 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-48) [3b8d5173] Correlation ID: 3b8d5173, Job ID: d045c7e4-638c-45f9-af29-f699a4e1e590, Call Stack: null, Custom Event ID: -1, Message: Storage Domain export (Data Center Default) was deactivated by system because it's not visible by any of the hosts. >2014-08-01 15:12:42,193 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-48) [3b8d5173] Removing vds [] from the domain in maintenance cache >2014-08-01 15:12:42,194 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-48) [3b8d5173] Removing host(s) [] from hosts unseen domain report cache >2014-08-01 15:12:45,293 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-34) [5484a050] START, GetHardwareInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]), log id: 60b6d538 >2014-08-01 15:12:45,346 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-34) [5484a050] FINISH, GetHardwareInfoVDSCommand, log id: 60b6d538 >2014-08-01 15:12:45,385 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-34) [26f56f7f] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS >2014-08-01 15:12:45,397 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-34) [28ec6d70] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-08-01 15:12:45,405 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-34) [44af4596] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-08-01 15:12:45,412 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-34) [44af4596] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, storageType = NFS, connectionList = [{ id: 05047c9a-152e-4ef2-b118-4a1753c1eae0, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35export, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 4a930483 >2014-08-01 15:12:45,471 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-34) [44af4596] FINISH, ConnectStorageServerVDSCommand, return: {5083f370-af09-457b-9b73-3440e8b65a07=0, 05047c9a-152e-4ef2-b118-4a1753c1eae0=0}, log id: 4a930483 >2014-08-01 15:12:45,472 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-34) [44af4596] Host rack storage connection was succeeded >2014-08-01 15:12:45,480 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-25) START, ConnectStoragePoolVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, masterVersion = 1), log id: 5225e95b >2014-08-01 15:12:52,001 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-6) [7f1e369e] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Inactive to Unknown, reason : no reporting hosts >2014-08-01 15:12:52,005 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-6) [7f1e369e] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Inactive to Unknown, reason : no reporting hosts >2014-08-01 15:13:45,879 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-25) FINISH, ConnectStoragePoolVDSCommand, log id: 5225e95b >2014-08-01 15:13:45,893 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-34) [44af4596] START, SetMOMPolicyParametersVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 45e0616 >2014-08-01 15:13:45,986 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-34) [44af4596] FINISH, SetMOMPolicyParametersVDSCommand, log id: 45e0616 >2014-08-01 15:13:45,994 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-34) [44af4596] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host rack was set to Up. >2014-08-01 15:13:45,999 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-34) [44af4596] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host rack. >2014-08-01 15:13:46,018 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-34) [36ef1be6] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS >2014-08-01 15:13:52,145 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-96) hostFromVds::selectedVds - rack, spmStatus Free, storage pool Default >2014-08-01 15:13:52,149 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-96) starting spm on vds rack, storage pool Default, prevId -1, LVER -1 >2014-08-01 15:13:52,151 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-96) START, SpmStartVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, prevId=-1, prevLVER=-1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 2ad314e6 >2014-08-01 15:13:52,200 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-96) spmStart polling started: taskId = 984f683a-d0fa-4c7b-9498-505542b322f1 >2014-08-01 15:13:53,207 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-96) spmStart polling ended: taskId = 984f683a-d0fa-4c7b-9498-505542b322f1 task status = finished >2014-08-01 15:13:53,262 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-96) spmStart polling ended, spm status: SPM >2014-08-01 15:13:53,264 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-96) START, HSMClearTaskVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, taskId=984f683a-d0fa-4c7b-9498-505542b322f1), log id: 258448fb >2014-08-01 15:13:53,309 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-96) FINISH, HSMClearTaskVDSCommand, log id: 258448fb >2014-08-01 15:13:53,310 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-96) FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@63cbbedf, log id: 2ad314e6 >2014-08-01 15:13:53,327 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-96) Initialize Irs proxy from vds: 10.34.62.208 >2014-08-01 15:13:53,345 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-96) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208). >2014-08-01 15:13:53,354 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-29) START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 5df37e2a >2014-08-01 15:13:53,395 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-29) -- executeIrsBrokerCommand: Attempting on storage pool 00000002-0002-0002-0002-0000000001d9 >2014-08-01 15:13:53,397 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-29) START, HSMGetAllTasksInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 7f4d8ddc >2014-08-01 15:13:53,402 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-29) FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 7f4d8ddc >2014-08-01 15:13:53,403 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-29) FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 5df37e2a >2014-08-01 15:13:53,404 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (org.ovirt.thread.pool-8-thread-29) Discovered no tasks on Storage Pool Default >2014-08-01 15:23:55,485 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-47) Principal admin::internal synchronized >2014-08-01 15:24:23,115 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-73) Attempting to update VMs/Templates Ovf. >2014-08-01 16:23:55,493 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-92) Principal admin::internal synchronized >2014-08-01 16:24:23,147 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-84) [7ef2d911] Attempting to update VMs/Templates Ovf. >2014-08-01 17:23:55,499 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-71) [6cd4f56d] Principal admin::internal synchronized >2014-08-01 17:24:23,163 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-73) Attempting to update VMs/Templates Ovf. >2014-08-01 18:13:40,434 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-46) domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 in problem. vds: rack >2014-08-01 18:13:40,436 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-46) domain 44559ec0-6bef-491c-bbd6-e67278f37987:export in problem. vds: rack >2014-08-01 18:15:11,463 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-54) Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2014-08-01 18:15:11,467 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-54) [14ab6114] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-08-01 18:15:11,471 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-54) [14ab6114] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Active to Unknown, reason : null >2014-08-01 18:15:11,473 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-54) [14ab6114] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Active to Unknown, reason : null >2014-08-01 18:15:11,488 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-54) [14ab6114] Correlation ID: 14ab6114, Call Stack: null, Custom Event ID: -1, Message: Invalid status on Data Center Default. Setting Data Center status to Non Responsive (On host rack, Error: Cannot access master storage domain). >2014-08-01 18:17:11,482 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-54) [14ab6114] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2014-08-01 18:17:11,484 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-54) [14ab6114] hostFromVds::selectedVds - rack, spmStatus returned null! >2014-08-01 18:17:11,486 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-54) [14ab6114] IrsBroker::Failed::GetStoragePoolInfoVDS >2014-08-01 18:17:11,486 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-54) [14ab6114] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2014-08-01 18:17:11,535 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-42) [668577b5] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage >2014-08-01 18:17:11,556 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-42) [668577b5] START, ResetIrsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, ignoreStopFailed = true), log id: 356feda2 >2014-08-01 18:17:11,559 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-42) [668577b5] START, SpmStopVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9), log id: 538ce19b >2014-08-01 18:17:11,580 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-42) [668577b5] SpmStopVDSCommand::Stopping SPM on vds rack, pool id 00000002-0002-0002-0002-0000000001d9 >2014-08-01 18:19:11,493 WARN [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-42) [668577b5] could not stop spm of pool 00000002-0002-0002-0002-0000000001d9 on vds 17f7fe56-5f32-40ef-b098-611b84484e0b - reason: org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcRunTimeException: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException >2014-08-01 18:19:11,495 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-42) [668577b5] FINISH, SpmStopVDSCommand, log id: 538ce19b >2014-08-01 18:19:11,498 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ListVDSCommand] (DefaultQuartzScheduler_Worker-15) [39225a1e] Command ListVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-08-01 18:19:11,501 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-15) [39225a1e] Host rack is not responding. It will stay in Connecting state for a grace period of $80 seconds and after that an attempt to fence the host will be issued. >2014-08-01 18:19:11,504 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-36) [3e2ffa8] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: VDSNetworkException: org.apache.commons.httpclient.NoHttpResponseException: The server 10.34.62.208 failed to respond >2014-08-01 18:19:11,502 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-42) [668577b5] FINISH, ResetIrsVDSCommand, log id: 356feda2 >2014-08-01 18:19:11,508 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-36) [3e2ffa8] hostFromVds::selectedVds - rack, spmStatus returned null! >2014-08-01 18:19:11,503 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-15) [39225a1e] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:406) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.fetchRunningVms(VdsUpdateRunTimeInfo.java:971) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVmStats(VdsUpdateRunTimeInfo.java:920) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:638) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:235) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) [:1.7.0_55] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_55] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_55] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_55] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_55] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_55] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_55] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_55] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy82.list(Unknown Source) > at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source) [:1.7.0_55] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_55] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_55] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_55] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_55] > >2014-08-01 18:19:11,522 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-42) [668577b5] Correlation ID: 668577b5, Job ID: ca102c87-ba6a-420b-bb83-e0fef82d5504, Call Stack: null, Custom Event ID: -1, Message: Failed to Reconstruct Master Domain for Data Center Default. >2014-08-01 18:19:11,545 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-15) [39225a1e] Failed to refresh VDS , vds = 17f7fe56-5f32-40ef-b098-611b84484e0b : rack, VDS Network Error, continuing. >java.net.ConnectException: Connection refused >2014-08-01 18:19:11,547 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-15) [14ab6114] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue >2014-08-01 18:19:11,549 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-46) starting processDomainRecovery for domain 44559ec0-6bef-491c-bbd6-e67278f37987:export >2014-08-01 18:19:11,617 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-46) Domain 44559ec0-6bef-491c-bbd6-e67278f37987:export was reported by all hosts in status UP as problematic. Moving the domain to NonOperational. >2014-08-01 18:19:11,643 INFO [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-46) [69558113] Lock Acquired to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE >, sharedLocks= ] >2014-08-01 18:19:11,650 INFO [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-46) [69558113] Running command: DeactivateStorageDomainCommand internal: true. Entities affected : ID: 44559ec0-6bef-491c-bbd6-e67278f37987 Type: Storage >2014-08-01 18:19:11,656 INFO [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-46) [69558113] Lock freed to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE >, sharedLocks= ] >2014-08-01 18:19:11,668 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-46) [69558113] Correlation ID: 69558113, Job ID: 0c702fcf-a0ce-4d1f-8178-de9a6ca40c0c, Call Stack: null, Custom Event ID: -1, Message: Storage Domain export (Data Center Default) was deactivated by system because it's not visible by any of the hosts. >2014-08-01 18:19:11,676 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-46) [69558113] Removing vds [] from the domain in maintenance cache >2014-08-01 18:19:11,677 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-46) [69558113] Removing host(s) [] from hosts unseen domain report cache >2014-08-01 18:19:11,678 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-23) starting processDomainRecovery for domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 >2014-08-01 18:19:11,684 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-23) Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 was reported by all hosts in status UP as problematic. Not moving the domain to NonOperational because it is being reconstructed now. >2014-08-01 18:19:11,701 WARN [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-23) [3d9b0e0c] CanDoAction of action ReconstructMasterDomain failed. Reasons:VAR__ACTION__RECONSTRUCT_MASTER,VAR__TYPE__STORAGE__DOMAIN,ACTION_TYPE_FAILED_NO_VDS_IN_POOL >2014-08-01 18:19:11,709 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-23) [3d9b0e0c] Removing vds [] from the domain in maintenance cache >2014-08-01 18:19:11,710 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-23) [3d9b0e0c] Removing host(s) [] from hosts unseen domain report cache >2014-08-01 18:19:11,711 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-15) [14ab6114] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue >2014-08-01 18:19:14,817 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-11) START, GetHardwareInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]), log id: ebb7847 >2014-08-01 18:19:14,869 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-11) FINISH, GetHardwareInfoVDSCommand, log id: ebb7847 >2014-08-01 18:19:14,895 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-11) [11d24648] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS >2014-08-01 18:19:14,909 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-11) [277b9d61] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-08-01 18:19:14,918 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-11) [3fb6f6e9] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-08-01 18:19:14,931 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-11) [3fb6f6e9] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, storageType = NFS, connectionList = [{ id: 05047c9a-152e-4ef2-b118-4a1753c1eae0, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35export, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 18d74e3 >2014-08-01 18:19:14,993 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-11) [3fb6f6e9] FINISH, ConnectStorageServerVDSCommand, return: {5083f370-af09-457b-9b73-3440e8b65a07=0, 05047c9a-152e-4ef2-b118-4a1753c1eae0=0}, log id: 18d74e3 >2014-08-01 18:19:14,994 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-11) [3fb6f6e9] Host rack storage connection was succeeded >2014-08-01 18:19:15,003 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-2) START, ConnectStoragePoolVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, masterVersion = 1), log id: 55f1b85c >2014-08-01 18:19:21,515 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-98) [5aa74089] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Inactive to Unknown, reason : no reporting hosts >2014-08-01 18:19:21,519 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-98) [5aa74089] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Inactive to Unknown, reason : no reporting hosts >2014-08-01 18:20:15,422 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-2) FINISH, ConnectStoragePoolVDSCommand, log id: 55f1b85c >2014-08-01 18:20:15,439 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-11) [3fb6f6e9] START, SetMOMPolicyParametersVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 36068ab9 >2014-08-01 18:20:15,531 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-11) [3fb6f6e9] FINISH, SetMOMPolicyParametersVDSCommand, log id: 36068ab9 >2014-08-01 18:20:15,538 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-11) [3fb6f6e9] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host rack was set to Up. >2014-08-01 18:20:15,545 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-11) [3fb6f6e9] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host rack. >2014-08-01 18:20:15,566 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-11) [aeb0141] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS >2014-08-01 18:20:21,666 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-16) hostFromVds::selectedVds - rack, spmStatus Free, storage pool Default >2014-08-01 18:20:21,672 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-16) starting spm on vds rack, storage pool Default, prevId -1, LVER -1 >2014-08-01 18:20:21,675 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-16) START, SpmStartVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, prevId=-1, prevLVER=-1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 12d526d5 >2014-08-01 18:20:21,723 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-16) spmStart polling started: taskId = 6f93fc93-89bd-42fb-897f-2baa1071d5c8 >2014-08-01 18:20:22,729 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-16) spmStart polling ended: taskId = 6f93fc93-89bd-42fb-897f-2baa1071d5c8 task status = finished >2014-08-01 18:20:22,785 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-16) spmStart polling ended, spm status: SPM >2014-08-01 18:20:22,788 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-16) START, HSMClearTaskVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, taskId=6f93fc93-89bd-42fb-897f-2baa1071d5c8), log id: 72b55b14 >2014-08-01 18:20:22,832 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-16) FINISH, HSMClearTaskVDSCommand, log id: 72b55b14 >2014-08-01 18:20:22,833 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-16) FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@6024726b, log id: 12d526d5 >2014-08-01 18:20:22,838 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-16) Initialize Irs proxy from vds: 10.34.62.208 >2014-08-01 18:20:22,845 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-16) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208). >2014-08-01 18:20:22,856 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-26) START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 465d34c5 >2014-08-01 18:20:22,886 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-26) -- executeIrsBrokerCommand: Attempting on storage pool 00000002-0002-0002-0002-0000000001d9 >2014-08-01 18:20:22,888 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-26) START, HSMGetAllTasksInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 1c9b88a6 >2014-08-01 18:20:22,894 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-26) FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 1c9b88a6 >2014-08-01 18:20:22,894 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-26) FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 465d34c5 >2014-08-01 18:20:22,896 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (org.ovirt.thread.pool-8-thread-26) Discovered no tasks on Storage Pool Default >2014-08-01 18:23:55,506 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-98) [5aa74089] Principal admin::internal synchronized >2014-08-01 18:24:23,180 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-86) Attempting to update VMs/Templates Ovf. >2014-08-01 19:23:55,512 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-84) [7ef2d911] Principal admin::internal synchronized >2014-08-01 19:24:23,211 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-26) [6bd9692] Attempting to update VMs/Templates Ovf. >2014-08-01 20:23:55,516 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-87) Principal admin::internal synchronized >2014-08-01 20:24:23,226 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-63) Attempting to update VMs/Templates Ovf. >2014-08-01 21:20:12,939 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-40) domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 in problem. vds: rack >2014-08-01 21:20:28,063 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-47) domain 44559ec0-6bef-491c-bbd6-e67278f37987:export in problem. vds: rack >2014-08-01 21:21:45,678 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-21) Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2014-08-01 21:21:45,682 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-21) [81f884c] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-08-01 21:21:45,687 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-21) [81f884c] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Active to Unknown, reason : null >2014-08-01 21:21:45,691 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-21) [81f884c] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Active to Unknown, reason : null >2014-08-01 21:21:45,704 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-21) [81f884c] Correlation ID: 81f884c, Call Stack: null, Custom Event ID: -1, Message: Invalid status on Data Center Default. Setting Data Center status to Non Responsive (On host rack, Error: Cannot access master storage domain). >2014-08-01 21:23:45,750 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-21) [81f884c] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2014-08-01 21:23:45,752 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-21) [81f884c] hostFromVds::selectedVds - rack, spmStatus returned null! >2014-08-01 21:23:45,754 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-21) [81f884c] IrsBroker::Failed::GetStoragePoolInfoVDS >2014-08-01 21:23:45,755 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-21) [81f884c] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2014-08-01 21:23:45,791 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-1) [5f8d1a83] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage >2014-08-01 21:23:45,810 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-1) [5f8d1a83] START, ResetIrsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, ignoreStopFailed = true), log id: e14837a >2014-08-01 21:23:45,816 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-1) [5f8d1a83] START, SpmStopVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9), log id: 653e7fef >2014-08-01 21:23:45,826 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-1) [5f8d1a83] SpmStopVDSCommand::Stopping SPM on vds rack, pool id 00000002-0002-0002-0002-0000000001d9 >2014-08-01 21:23:55,520 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-69) [52d379dc] Principal admin::internal synchronized >2014-08-01 21:24:23,240 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-52) Attempting to update VMs/Templates Ovf. >2014-08-01 21:25:00,007 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-62) [2e742a17] Autorecovering 1 storage domains >2014-08-01 21:25:00,008 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-62) [2e742a17] Autorecovering storage domains id: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 >2014-08-01 21:25:00,012 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-62) [2cdbf22e] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage >2014-08-01 21:25:00,014 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-62) [2cdbf22e] ConnectDomainToStorage. Before Connect all hosts to pool. Time:8/1/14 9:25 PM >2014-08-01 21:25:00,024 INFO [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-8-thread-49) [2b4c7004] Running command: ConnectStorageToVdsCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2014-08-01 21:25:00,029 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-49) [2b4c7004] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 38b92ae0 >2014-08-01 21:25:45,713 WARN [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-1) [5f8d1a83] could not stop spm of pool 00000002-0002-0002-0002-0000000001d9 on vds 17f7fe56-5f32-40ef-b098-611b84484e0b - reason: org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcRunTimeException: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException >2014-08-01 21:25:45,715 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-1) [5f8d1a83] FINISH, SpmStopVDSCommand, log id: 653e7fef >2014-08-01 21:25:45,720 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-88) Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: VDSNetworkException: org.apache.commons.httpclient.NoHttpResponseException: The server 10.34.62.208 failed to respond >2014-08-01 21:25:45,720 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ListVDSCommand] (DefaultQuartzScheduler_Worker-28) [8ef1638] Command ListVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-08-01 21:25:45,719 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-49) [2b4c7004] Command ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-08-01 21:25:45,725 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-28) [8ef1638] Host rack is not responding. It will stay in Connecting state for a grace period of $80 seconds and after that an attempt to fence the host will be issued. >2014-08-01 21:25:45,730 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-1) [5f8d1a83] FINISH, ResetIrsVDSCommand, log id: e14837a >2014-08-01 21:25:45,724 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-88) hostFromVds::selectedVds - rack, spmStatus returned null! >2014-08-01 21:25:45,728 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-49) [2b4c7004] FINISH, ConnectStorageServerVDSCommand, log id: 38b92ae0 >2014-08-01 21:25:45,733 ERROR [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-8-thread-49) [2b4c7004] Command org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-08-01 21:25:45,730 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-28) [8ef1638] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:406) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.fetchRunningVms(VdsUpdateRunTimeInfo.java:971) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVmStats(VdsUpdateRunTimeInfo.java:920) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:638) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:235) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) [:1.7.0_55] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_55] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_55] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_55] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_55] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_55] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_55] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_55] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy82.list(Unknown Source) > at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source) [:1.7.0_55] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_55] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_55] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_55] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_55] > >2014-08-01 21:25:45,746 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-1) [5f8d1a83] Correlation ID: 5f8d1a83, Job ID: e1c97138-8cf1-4fb6-8a9d-5e9e4910f98d, Call Stack: null, Custom Event ID: -1, Message: Failed to Reconstruct Master Domain for Data Center Default. >2014-08-01 21:25:45,737 ERROR [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-8-thread-49) [2b4c7004] Transaction rolled-back for command: org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand. >2014-08-01 21:25:45,767 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-62) [2cdbf22e] ConnectDomainToStorage. After Connect all hosts to pool. Time:8/1/14 9:25 PM >2014-08-01 21:25:45,771 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-42) [81f884c] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue >2014-08-01 21:25:45,773 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-28) [8ef1638] Failed to refresh VDS , vds = 17f7fe56-5f32-40ef-b098-611b84484e0b : rack, VDS Network Error, continuing. >java.net.ConnectException: Connection refused >2014-08-01 21:25:45,774 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-5) starting processDomainRecovery for domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 >2014-08-01 21:25:45,825 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-5) Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 was reported by all hosts in status UP as problematic. Not moving the domain to NonOperational because it is being reconstructed now. >2014-08-01 21:25:45,843 WARN [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-5) [6edda802] CanDoAction of action ReconstructMasterDomain failed. Reasons:VAR__ACTION__RECONSTRUCT_MASTER,VAR__TYPE__STORAGE__DOMAIN,ACTION_TYPE_FAILED_NO_VDS_IN_POOL >2014-08-01 21:25:45,850 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-5) [6edda802] Removing vds [] from the domain in maintenance cache >2014-08-01 21:25:45,851 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-5) [6edda802] Removing host(s) [] from hosts unseen domain report cache >2014-08-01 21:25:45,852 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-42) [81f884c] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue >2014-08-01 21:25:45,853 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-17) starting processDomainRecovery for domain 44559ec0-6bef-491c-bbd6-e67278f37987:export >2014-08-01 21:25:45,858 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-17) Domain 44559ec0-6bef-491c-bbd6-e67278f37987:export was reported by all hosts in status UP as problematic. Moving the domain to NonOperational. >2014-08-01 21:25:45,872 INFO [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-17) [41751a9b] Lock Acquired to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE >, sharedLocks= ] >2014-08-01 21:25:45,878 INFO [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-17) [41751a9b] Running command: DeactivateStorageDomainCommand internal: true. Entities affected : ID: 44559ec0-6bef-491c-bbd6-e67278f37987 Type: Storage >2014-08-01 21:25:45,883 INFO [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-17) [41751a9b] Lock freed to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE >, sharedLocks= ] >2014-08-01 21:25:45,892 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-17) [41751a9b] Correlation ID: 41751a9b, Job ID: cd8ae9ed-6d97-4de3-a4a9-6c1a40360401, Call Stack: null, Custom Event ID: -1, Message: Storage Domain export (Data Center Default) was deactivated by system because it's not visible by any of the hosts. >2014-08-01 21:25:45,897 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-17) [41751a9b] Removing vds [] from the domain in maintenance cache >2014-08-01 21:25:45,897 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-17) [41751a9b] Removing host(s) [] from hosts unseen domain report cache >2014-08-01 21:25:49,039 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-91) START, GetHardwareInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]), log id: 5aba8df1 >2014-08-01 21:25:49,094 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-91) FINISH, GetHardwareInfoVDSCommand, log id: 5aba8df1 >2014-08-01 21:25:49,125 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-91) [158bf392] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS >2014-08-01 21:25:49,138 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-91) [4e7820e7] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-08-01 21:25:49,145 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-91) [3721c63a] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-08-01 21:25:49,151 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-91) [3721c63a] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, storageType = NFS, connectionList = [{ id: 05047c9a-152e-4ef2-b118-4a1753c1eae0, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35export, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 5208656 >2014-08-01 21:25:49,213 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-91) [3721c63a] FINISH, ConnectStorageServerVDSCommand, return: {5083f370-af09-457b-9b73-3440e8b65a07=0, 05047c9a-152e-4ef2-b118-4a1753c1eae0=0}, log id: 5208656 >2014-08-01 21:25:49,214 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-91) [3721c63a] Host rack storage connection was succeeded >2014-08-01 21:25:49,220 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-11) START, ConnectStoragePoolVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, masterVersion = 1), log id: 76fa79 >2014-08-01 21:25:55,736 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-23) [797bf730] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Inactive to Unknown, reason : no reporting hosts >2014-08-01 21:25:55,740 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-23) [797bf730] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Inactive to Unknown, reason : no reporting hosts >2014-08-01 21:26:49,604 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-11) FINISH, ConnectStoragePoolVDSCommand, log id: 76fa79 >2014-08-01 21:26:49,618 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-91) [3721c63a] START, SetMOMPolicyParametersVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 7a550198 >2014-08-01 21:26:49,711 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-91) [3721c63a] FINISH, SetMOMPolicyParametersVDSCommand, log id: 7a550198 >2014-08-01 21:26:49,721 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-91) [3721c63a] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host rack was set to Up. >2014-08-01 21:26:49,728 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-91) [3721c63a] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host rack. >2014-08-01 21:26:49,755 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-91) [64566e87] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS >2014-08-01 21:26:55,867 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-15) [39225a1e] hostFromVds::selectedVds - rack, spmStatus Free, storage pool Default >2014-08-01 21:26:55,873 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-15) [39225a1e] starting spm on vds rack, storage pool Default, prevId -1, LVER -1 >2014-08-01 21:26:55,875 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-15) [39225a1e] START, SpmStartVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, prevId=-1, prevLVER=-1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 5e71842e >2014-08-01 21:26:55,924 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-15) [39225a1e] spmStart polling started: taskId = c08e9839-c965-4fab-97c4-08aad7ae46a0 >2014-08-01 21:26:56,930 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-15) [39225a1e] spmStart polling ended: taskId = c08e9839-c965-4fab-97c4-08aad7ae46a0 task status = finished >2014-08-01 21:26:56,986 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-15) [39225a1e] spmStart polling ended, spm status: SPM >2014-08-01 21:26:56,989 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-15) [39225a1e] START, HSMClearTaskVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, taskId=c08e9839-c965-4fab-97c4-08aad7ae46a0), log id: 21087e37 >2014-08-01 21:26:57,035 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-15) [39225a1e] FINISH, HSMClearTaskVDSCommand, log id: 21087e37 >2014-08-01 21:26:57,036 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-15) [39225a1e] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@1358db5d, log id: 5e71842e >2014-08-01 21:26:57,040 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-15) [39225a1e] Initialize Irs proxy from vds: 10.34.62.208 >2014-08-01 21:26:57,046 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-15) [39225a1e] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208). >2014-08-01 21:26:57,048 WARN [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-8-thread-14) Executing a command: java.util.concurrent.FutureTask , but note that there are 1 tasks in the queue. >2014-08-01 21:26:57,062 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-14) [39225a1e] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 46968ee2 >2014-08-01 21:26:57,085 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-14) [39225a1e] -- executeIrsBrokerCommand: Attempting on storage pool 00000002-0002-0002-0002-0000000001d9 >2014-08-01 21:26:57,087 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-14) [39225a1e] START, HSMGetAllTasksInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 6e39348c >2014-08-01 21:26:57,092 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-14) [39225a1e] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 6e39348c >2014-08-01 21:26:57,093 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-14) [39225a1e] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 46968ee2 >2014-08-01 21:26:57,094 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (org.ovirt.thread.pool-8-thread-14) [39225a1e] Discovered no tasks on Storage Pool Default >2014-08-01 22:23:55,526 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-60) [1cb6c42c] Principal admin::internal synchronized >2014-08-01 22:24:23,243 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-80) Attempting to update VMs/Templates Ovf. >2014-08-01 23:23:55,532 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-63) Principal admin::internal synchronized >2014-08-01 23:24:23,273 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-58) [7b19edd8] Attempting to update VMs/Templates Ovf. >2014-08-02 00:23:55,538 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-49) Principal admin::internal synchronized >2014-08-02 00:24:23,290 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-90) Attempting to update VMs/Templates Ovf. >2014-08-02 00:26:46,585 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-49) domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 in problem. vds: rack >2014-08-02 00:27:01,710 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-7) domain 44559ec0-6bef-491c-bbd6-e67278f37987:export in problem. vds: rack >2014-08-02 00:28:17,827 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-75) [52dee25c] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2014-08-02 00:28:17,832 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-75) [6e437c9b] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-08-02 00:28:17,835 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-75) [6e437c9b] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Active to Unknown, reason : null >2014-08-02 00:28:17,837 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-75) [6e437c9b] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Active to Unknown, reason : null >2014-08-02 00:28:17,850 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-75) [6e437c9b] Correlation ID: 6e437c9b, Call Stack: null, Custom Event ID: -1, Message: Invalid status on Data Center Default. Setting Data Center status to Non Responsive (On host rack, Error: Cannot access master storage domain). >2014-08-02 00:30:17,877 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-75) [6e437c9b] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2014-08-02 00:30:17,880 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-75) [6e437c9b] hostFromVds::selectedVds - rack, spmStatus returned null! >2014-08-02 00:30:17,881 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-75) [6e437c9b] IrsBroker::Failed::GetStoragePoolInfoVDS >2014-08-02 00:30:17,882 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-75) [6e437c9b] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2014-08-02 00:30:17,931 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-14) [3aa22b25] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage >2014-08-02 00:30:17,951 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-14) [3aa22b25] START, ResetIrsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, ignoreStopFailed = true), log id: 2d74bbdd >2014-08-02 00:30:17,980 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-14) [3aa22b25] START, SpmStopVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9), log id: 680f5ec5 >2014-08-02 00:30:17,987 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-14) [3aa22b25] SpmStopVDSCommand::Stopping SPM on vds rack, pool id 00000002-0002-0002-0002-0000000001d9 >2014-08-02 00:32:17,854 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-15) [39225a1e] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2014-08-02 00:32:17,857 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-15) [39225a1e] hostFromVds::selectedVds - rack, spmStatus returned null! >2014-08-02 00:32:17,858 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-15) [39225a1e] IrsBroker::Failed::GetStoragePoolInfoVDS >2014-08-02 00:32:17,858 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-15) [39225a1e] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2014-08-02 00:33:17,989 WARN [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-14) [3aa22b25] could not stop spm of pool 00000002-0002-0002-0002-0000000001d9 on vds 17f7fe56-5f32-40ef-b098-611b84484e0b - reason: org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcRunTimeException: java.util.concurrent.TimeoutException >2014-08-02 00:33:17,990 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-14) [3aa22b25] FINISH, SpmStopVDSCommand, log id: 680f5ec5 >2014-08-02 00:33:17,997 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-14) [3aa22b25] FINISH, ResetIrsVDSCommand, log id: 2d74bbdd >2014-08-02 00:33:18,007 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-14) [3aa22b25] Correlation ID: 3aa22b25, Job ID: 5cc7a651-da12-49a9-ac41-77cca53d54ba, Call Stack: null, Custom Event ID: -1, Message: Failed to Reconstruct Master Domain for Data Center Default. >2014-08-02 00:33:18,015 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-11) [6e437c9b] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue >2014-08-02 00:33:18,017 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-17) starting processDomainRecovery for domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 >2014-08-02 00:33:18,073 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-17) Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 was reported by all hosts in status UP as problematic. Not moving the domain to NonOperational because it is being reconstructed now. >2014-08-02 00:33:18,128 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-17) [b275d22] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage >2014-08-02 00:33:18,158 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-17) [b275d22] Removing vds [] from the domain in maintenance cache >2014-08-02 00:33:18,159 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-17) [b275d22] Removing host(s) [] from hosts unseen domain report cache >2014-08-02 00:33:18,160 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-11) [6e437c9b] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue >2014-08-02 00:33:18,162 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-48) starting processDomainRecovery for domain 44559ec0-6bef-491c-bbd6-e67278f37987:export >2014-08-02 00:33:18,167 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-48) Domain 44559ec0-6bef-491c-bbd6-e67278f37987:export was reported by all hosts in status UP as problematic. Moving the domain to NonOperational. >2014-08-02 00:33:18,182 INFO [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-48) [688ba99] Lock Acquired to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE >, sharedLocks= ] >2014-08-02 00:33:18,188 INFO [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-48) [688ba99] Running command: DeactivateStorageDomainCommand internal: true. Entities affected : ID: 44559ec0-6bef-491c-bbd6-e67278f37987 Type: Storage >2014-08-02 00:33:18,195 INFO [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-48) [688ba99] Lock freed to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE >, sharedLocks= ] >2014-08-02 00:33:18,205 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-48) [688ba99] Correlation ID: 688ba99, Job ID: 7231f96a-616b-4d52-8c57-710db6b5a94b, Call Stack: null, Custom Event ID: -1, Message: Storage Domain export (Data Center Default) was deactivated by system because it's not visible by any of the hosts. >2014-08-02 00:33:18,213 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-48) [688ba99] Removing vds [] from the domain in maintenance cache >2014-08-02 00:33:18,214 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-48) [688ba99] Removing host(s) [] from hosts unseen domain report cache >2014-08-02 00:34:17,868 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-31) Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: VDSNetworkException: org.apache.commons.httpclient.NoHttpResponseException: The server 10.34.62.208 failed to respond >2014-08-02 00:34:17,870 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-31) hostFromVds::selectedVds - rack, spmStatus returned null! >2014-08-02 00:34:18,752 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ListVDSCommand] (DefaultQuartzScheduler_Worker-94) Command ListVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]) execution failed. Exception: VDSRecoveringException: Recovering from crash or Initializing >2014-08-02 00:34:18,753 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-94) Failed vds listing, vds = 17f7fe56-5f32-40ef-b098-611b84484e0b : rack, error = org.ovirt.engine.core.vdsbroker.vdsbroker.VDSRecoveringException: Recovering from crash or Initializing >2014-08-02 00:34:18,766 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-94) Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSRecoveringException: Recovering from crash or Initializing > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.vdsbroker.ListVDSCommand.executeVdsBrokerCommand(ListVDSCommand.java:24) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:406) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.fetchRunningVms(VdsUpdateRunTimeInfo.java:971) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVmStats(VdsUpdateRunTimeInfo.java:920) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:638) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:235) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Host rack is initializing. Message: Recovering from crash or Initializing >2014-08-02 00:34:18,775 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-94) Failed to refresh VDS , vds = 17f7fe56-5f32-40ef-b098-611b84484e0b : rack, error = Recovering from crash or Initializing, continuing. >2014-08-02 00:34:22,013 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-23) [797bf730] START, GetHardwareInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]), log id: 83765ef >2014-08-02 00:34:22,066 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-23) [797bf730] FINISH, GetHardwareInfoVDSCommand, log id: 83765ef >2014-08-02 00:34:22,093 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-23) [632bb53a] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS >2014-08-02 00:34:22,142 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-23) [632bb53a] Cancelling the recovery from crash timer for VDS 17f7fe56-5f32-40ef-b098-611b84484e0b because vds started initializing >2014-08-02 00:34:22,144 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-23) [65662402] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-08-02 00:34:22,152 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-23) [5f6025d1] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-08-02 00:34:22,160 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-23) [5f6025d1] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, storageType = NFS, connectionList = [{ id: 05047c9a-152e-4ef2-b118-4a1753c1eae0, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35export, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 6a0f7872 >2014-08-02 00:34:22,226 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-23) [5f6025d1] FINISH, ConnectStorageServerVDSCommand, return: {5083f370-af09-457b-9b73-3440e8b65a07=0, 05047c9a-152e-4ef2-b118-4a1753c1eae0=0}, log id: 6a0f7872 >2014-08-02 00:34:22,227 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-23) [5f6025d1] Host rack storage connection was succeeded >2014-08-02 00:34:22,234 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-15) START, ConnectStoragePoolVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, masterVersion = 1), log id: 3ea3093e >2014-08-02 00:34:27,874 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-39) [4a2f7e33] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Inactive to Unknown, reason : no reporting hosts >2014-08-02 00:34:27,878 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-39) [4a2f7e33] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Inactive to Unknown, reason : no reporting hosts >2014-08-02 00:35:22,627 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-15) FINISH, ConnectStoragePoolVDSCommand, log id: 3ea3093e >2014-08-02 00:35:22,640 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-23) [5f6025d1] START, SetMOMPolicyParametersVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 5250ec7d >2014-08-02 00:35:22,734 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-23) [5f6025d1] FINISH, SetMOMPolicyParametersVDSCommand, log id: 5250ec7d >2014-08-02 00:35:22,741 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-23) [5f6025d1] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host rack was set to Up. >2014-08-02 00:35:22,747 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-23) [5f6025d1] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host rack. >2014-08-02 00:35:22,763 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-23) [6169221c] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS >2014-08-02 00:35:27,989 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-3) hostFromVds::selectedVds - rack, spmStatus Free, storage pool Default >2014-08-02 00:35:27,994 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-3) starting spm on vds rack, storage pool Default, prevId -1, LVER -1 >2014-08-02 00:35:27,996 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-3) START, SpmStartVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, prevId=-1, prevLVER=-1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 5d4d1318 >2014-08-02 00:35:28,044 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-3) spmStart polling started: taskId = be9a2371-7b04-4133-a3e1-599eda518827 >2014-08-02 00:35:29,053 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-3) spmStart polling ended: taskId = be9a2371-7b04-4133-a3e1-599eda518827 task status = finished >2014-08-02 00:35:29,109 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-3) spmStart polling ended, spm status: SPM >2014-08-02 00:35:29,111 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-3) START, HSMClearTaskVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, taskId=be9a2371-7b04-4133-a3e1-599eda518827), log id: 218f54dc >2014-08-02 00:35:29,156 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-3) FINISH, HSMClearTaskVDSCommand, log id: 218f54dc >2014-08-02 00:35:29,158 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-3) FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@206ea66a, log id: 5d4d1318 >2014-08-02 00:35:29,161 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-3) Initialize Irs proxy from vds: 10.34.62.208 >2014-08-02 00:35:29,168 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-3) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208). >2014-08-02 00:35:29,169 WARN [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-8-thread-24) Executing a command: java.util.concurrent.FutureTask , but note that there are 1 tasks in the queue. >2014-08-02 00:35:29,177 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-24) START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 5680d7b9 >2014-08-02 00:35:29,220 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-24) -- executeIrsBrokerCommand: Attempting on storage pool 00000002-0002-0002-0002-0000000001d9 >2014-08-02 00:35:29,222 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-24) START, HSMGetAllTasksInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 58a1d10 >2014-08-02 00:35:29,228 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-24) FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 58a1d10 >2014-08-02 00:35:29,229 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-24) FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 5680d7b9 >2014-08-02 00:35:29,230 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (org.ovirt.thread.pool-8-thread-24) Discovered no tasks on Storage Pool Default >2014-08-02 00:56:36,377 WARN [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-8-thread-39) Executing a command: java.util.concurrent.FutureTask , but note that there are 1 tasks in the queue. >2014-08-02 01:23:55,542 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-84) [7ef2d911] Principal admin::internal synchronized >2014-08-02 01:24:23,306 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-47) Attempting to update VMs/Templates Ovf. >2014-08-02 02:23:55,551 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-71) [6cd4f56d] Principal admin::internal synchronized >2014-08-02 02:24:23,324 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-81) Attempting to update VMs/Templates Ovf. >2014-08-02 03:23:55,560 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-46) [56f1581c] Principal admin::internal synchronized >2014-08-02 03:24:23,356 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-11) [aeb0141] Attempting to update VMs/Templates Ovf. >2014-08-02 03:35:08,495 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-28) domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 in problem. vds: rack >2014-08-02 03:35:23,623 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-38) domain 44559ec0-6bef-491c-bbd6-e67278f37987:export in problem. vds: rack >2014-08-02 03:35:35,001 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-88) Start deleteAgedOutAuditLogs >2014-08-02 03:35:35,002 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-38) [1b9ef0f] Start deleteAgedOutCommandEntities >2014-08-02 03:35:35,007 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-88) Finished deleteAgedOutAuditLogs >2014-08-02 03:35:35,010 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-38) [1b9ef0f] Finished deleteAgedOutCommandEntities >2014-08-02 03:36:48,100 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-86) Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2014-08-02 03:36:48,105 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-86) [6452f878] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-08-02 03:36:48,108 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-86) [6452f878] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Active to Unknown, reason : null >2014-08-02 03:36:48,110 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-86) [6452f878] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Active to Unknown, reason : null >2014-08-02 03:36:48,125 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-86) [6452f878] Correlation ID: 6452f878, Call Stack: null, Custom Event ID: -1, Message: Invalid status on Data Center Default. Setting Data Center status to Non Responsive (On host rack, Error: Cannot access master storage domain). >2014-08-02 03:38:48,165 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-86) [6452f878] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2014-08-02 03:38:48,167 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-86) [6452f878] hostFromVds::selectedVds - rack, spmStatus returned null! >2014-08-02 03:38:48,168 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-86) [6452f878] IrsBroker::Failed::GetStoragePoolInfoVDS >2014-08-02 03:38:48,169 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-86) [6452f878] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2014-08-02 03:38:48,209 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-44) [31614d3b] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage >2014-08-02 03:38:48,230 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-44) [31614d3b] START, ResetIrsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, ignoreStopFailed = true), log id: 6aa9a5ad >2014-08-02 03:38:48,233 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-44) [31614d3b] START, SpmStopVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9), log id: 79d6077b >2014-08-02 03:38:48,245 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-44) [31614d3b] SpmStopVDSCommand::Stopping SPM on vds rack, pool id 00000002-0002-0002-0002-0000000001d9 >2014-08-02 03:40:00,005 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-33) Autorecovering 1 storage domains >2014-08-02 03:40:00,006 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-33) Autorecovering storage domains id: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 >2014-08-02 03:40:00,008 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-33) [561d02fb] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage >2014-08-02 03:40:00,010 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-33) [561d02fb] ConnectDomainToStorage. Before Connect all hosts to pool. Time:8/2/14 3:40 AM >2014-08-02 03:40:00,019 INFO [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-8-thread-36) [6cd0eb4a] Running command: ConnectStorageToVdsCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2014-08-02 03:40:00,023 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-36) [6cd0eb4a] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 56ca28f >2014-08-02 03:40:48,152 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-80) Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2014-08-02 03:40:48,154 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-80) hostFromVds::selectedVds - rack, spmStatus returned null! >2014-08-02 03:40:48,155 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-80) IrsBroker::Failed::GetStoragePoolInfoVDS >2014-08-02 03:40:48,156 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-80) Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2014-08-02 03:41:48,151 WARN [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-44) [31614d3b] could not stop spm of pool 00000002-0002-0002-0002-0000000001d9 on vds 17f7fe56-5f32-40ef-b098-611b84484e0b - reason: org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcRunTimeException: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException >2014-08-02 03:41:48,155 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-44) [31614d3b] FINISH, SpmStopVDSCommand, log id: 79d6077b >2014-08-02 03:41:48,154 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-36) [6cd0eb4a] Command ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]) execution failed. Exception: VDSNetworkException: org.apache.commons.httpclient.NoHttpResponseException: The server 10.34.62.208 failed to respond >2014-08-02 03:41:48,160 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ListVDSCommand] (DefaultQuartzScheduler_Worker-21) [81f884c] Command ListVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-08-02 03:41:48,158 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-27) Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-08-02 03:41:48,167 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-21) [81f884c] Host rack is not responding. It will stay in Connecting state for a grace period of $80 seconds and after that an attempt to fence the host will be issued. >2014-08-02 03:41:48,163 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-36) [6cd0eb4a] FINISH, ConnectStorageServerVDSCommand, log id: 56ca28f >2014-08-02 03:41:48,173 ERROR [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-8-thread-36) [6cd0eb4a] Command org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: org.apache.commons.httpclient.NoHttpResponseException: The server 10.34.62.208 failed to respond (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-08-02 03:41:48,171 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-44) [31614d3b] FINISH, ResetIrsVDSCommand, log id: 6aa9a5ad >2014-08-02 03:41:48,169 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-27) hostFromVds::selectedVds - rack, spmStatus returned null! >2014-08-02 03:41:48,171 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-21) [81f884c] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:406) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.fetchRunningVms(VdsUpdateRunTimeInfo.java:971) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVmStats(VdsUpdateRunTimeInfo.java:920) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:638) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:235) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) [:1.7.0_55] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_55] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_55] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_55] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_55] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_55] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_55] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_55] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy82.list(Unknown Source) > at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source) [:1.7.0_55] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_55] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_55] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_55] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_55] > >2014-08-02 03:41:48,190 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-44) [31614d3b] Correlation ID: 31614d3b, Job ID: 126a434b-1c98-4772-9e8d-1e36173331fd, Call Stack: null, Custom Event ID: -1, Message: Failed to Reconstruct Master Domain for Data Center Default. >2014-08-02 03:41:48,182 ERROR [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-8-thread-36) [6cd0eb4a] Transaction rolled-back for command: org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand. >2014-08-02 03:41:48,213 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-33) [561d02fb] ConnectDomainToStorage. After Connect all hosts to pool. Time:8/2/14 3:41 AM >2014-08-02 03:41:48,219 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-21) [81f884c] Failed to refresh VDS , vds = 17f7fe56-5f32-40ef-b098-611b84484e0b : rack, VDS Network Error, continuing. >java.net.ConnectException: Connection refused >2014-08-02 03:41:48,221 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-24) [6452f878] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue >2014-08-02 03:41:48,223 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-38) starting processDomainRecovery for domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 >2014-08-02 03:41:48,285 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-38) Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 was reported by all hosts in status UP as problematic. Not moving the domain to NonOperational because it is being reconstructed now. >2014-08-02 03:41:48,303 WARN [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-38) [48543fa4] CanDoAction of action ReconstructMasterDomain failed. Reasons:VAR__ACTION__RECONSTRUCT_MASTER,VAR__TYPE__STORAGE__DOMAIN,ACTION_TYPE_FAILED_NO_VDS_IN_POOL >2014-08-02 03:41:48,310 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-38) [48543fa4] Removing vds [] from the domain in maintenance cache >2014-08-02 03:41:48,311 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-38) [48543fa4] Removing host(s) [] from hosts unseen domain report cache >2014-08-02 03:41:48,312 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-24) [6452f878] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue >2014-08-02 03:41:48,314 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-43) starting processDomainRecovery for domain 44559ec0-6bef-491c-bbd6-e67278f37987:export >2014-08-02 03:41:48,319 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-43) Domain 44559ec0-6bef-491c-bbd6-e67278f37987:export was reported by all hosts in status UP as problematic. Moving the domain to NonOperational. >2014-08-02 03:41:48,333 INFO [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-43) [159b9156] Lock Acquired to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE >, sharedLocks= ] >2014-08-02 03:41:48,340 INFO [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-43) [159b9156] Running command: DeactivateStorageDomainCommand internal: true. Entities affected : ID: 44559ec0-6bef-491c-bbd6-e67278f37987 Type: Storage >2014-08-02 03:41:48,346 INFO [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-43) [159b9156] Lock freed to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE >, sharedLocks= ] >2014-08-02 03:41:48,356 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-43) [159b9156] Correlation ID: 159b9156, Job ID: ab9af258-7dea-45e2-8fae-f85d922dfae0, Call Stack: null, Custom Event ID: -1, Message: Storage Domain export (Data Center Default) was deactivated by system because it's not visible by any of the hosts. >2014-08-02 03:41:48,361 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-43) [159b9156] Removing vds [] from the domain in maintenance cache >2014-08-02 03:41:48,362 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-43) [159b9156] Removing host(s) [] from hosts unseen domain report cache >2014-08-02 03:41:51,470 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-82) [3abdaaa6] START, GetHardwareInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]), log id: ac91fab >2014-08-02 03:41:51,526 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-82) [3abdaaa6] FINISH, GetHardwareInfoVDSCommand, log id: ac91fab >2014-08-02 03:41:51,551 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-82) [50676e63] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS >2014-08-02 03:41:51,600 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-82) [5f8d8487] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-08-02 03:41:51,608 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-82) [f27a43f] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-08-02 03:41:51,620 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-82) [f27a43f] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, storageType = NFS, connectionList = [{ id: 05047c9a-152e-4ef2-b118-4a1753c1eae0, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35export, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 45b24c41 >2014-08-02 03:41:51,678 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-82) [f27a43f] FINISH, ConnectStorageServerVDSCommand, return: {5083f370-af09-457b-9b73-3440e8b65a07=0, 05047c9a-152e-4ef2-b118-4a1753c1eae0=0}, log id: 45b24c41 >2014-08-02 03:41:51,679 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-82) [f27a43f] Host rack storage connection was succeeded >2014-08-02 03:41:51,688 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-14) START, ConnectStoragePoolVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, masterVersion = 1), log id: 5acd2e98 >2014-08-02 03:41:58,184 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-83) Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Inactive to Unknown, reason : no reporting hosts >2014-08-02 03:41:58,188 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-83) Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Inactive to Unknown, reason : no reporting hosts >2014-08-02 03:42:52,056 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-14) FINISH, ConnectStoragePoolVDSCommand, log id: 5acd2e98 >2014-08-02 03:42:52,097 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-82) [f27a43f] START, SetMOMPolicyParametersVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 4ed6dec >2014-08-02 03:42:52,191 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-82) [f27a43f] FINISH, SetMOMPolicyParametersVDSCommand, log id: 4ed6dec >2014-08-02 03:42:52,198 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-82) [f27a43f] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host rack was set to Up. >2014-08-02 03:42:52,205 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-82) [f27a43f] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host rack. >2014-08-02 03:42:52,224 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-82) [3d60c3e4] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS >2014-08-02 03:42:58,367 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-50) [375fcc6] hostFromVds::selectedVds - rack, spmStatus Free, storage pool Default >2014-08-02 03:42:58,372 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-50) [375fcc6] starting spm on vds rack, storage pool Default, prevId -1, LVER -1 >2014-08-02 03:42:58,375 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-50) [375fcc6] START, SpmStartVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, prevId=-1, prevLVER=-1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 3ee604ee >2014-08-02 03:42:58,424 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-50) [375fcc6] spmStart polling started: taskId = 428f2bfa-2ea9-4e5d-9e82-f6b6e3c3b634 >2014-08-02 03:42:59,430 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-50) [375fcc6] spmStart polling ended: taskId = 428f2bfa-2ea9-4e5d-9e82-f6b6e3c3b634 task status = finished >2014-08-02 03:42:59,486 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-50) [375fcc6] spmStart polling ended, spm status: SPM >2014-08-02 03:42:59,488 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-50) [375fcc6] START, HSMClearTaskVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, taskId=428f2bfa-2ea9-4e5d-9e82-f6b6e3c3b634), log id: 3740354e >2014-08-02 03:42:59,534 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-50) [375fcc6] FINISH, HSMClearTaskVDSCommand, log id: 3740354e >2014-08-02 03:42:59,535 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-50) [375fcc6] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@885a4ff, log id: 3ee604ee >2014-08-02 03:42:59,539 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-50) [375fcc6] Initialize Irs proxy from vds: 10.34.62.208 >2014-08-02 03:42:59,546 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-50) [375fcc6] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208). >2014-08-02 03:42:59,548 WARN [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-8-thread-32) Executing a command: java.util.concurrent.FutureTask , but note that there are 1 tasks in the queue. >2014-08-02 03:42:59,568 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-32) [375fcc6] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 65eaae20 >2014-08-02 03:42:59,592 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-32) [375fcc6] -- executeIrsBrokerCommand: Attempting on storage pool 00000002-0002-0002-0002-0000000001d9 >2014-08-02 03:42:59,594 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-32) [375fcc6] START, HSMGetAllTasksInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 7938381d >2014-08-02 03:42:59,600 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-32) [375fcc6] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 7938381d >2014-08-02 03:42:59,601 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-32) [375fcc6] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 65eaae20 >2014-08-02 03:42:59,602 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (org.ovirt.thread.pool-8-thread-32) [375fcc6] Discovered no tasks on Storage Pool Default >2014-08-02 04:23:55,565 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-55) [7526d482] Principal admin::internal synchronized >2014-08-02 04:24:23,374 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-32) Attempting to update VMs/Templates Ovf. >2014-08-02 05:23:55,571 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-96) Principal admin::internal synchronized >2014-08-02 05:24:23,387 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-11) [aeb0141] Attempting to update VMs/Templates Ovf. >2014-08-02 06:23:55,585 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-27) Principal admin::internal synchronized >2014-08-02 06:24:23,401 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-52) Attempting to update VMs/Templates Ovf. >2014-08-02 06:26:17,467 WARN [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-8-thread-18) Executing a command: java.util.concurrent.FutureTask , but note that there are 1 tasks in the queue. >2014-08-02 06:42:40,616 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-35) domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 in problem. vds: rack >2014-08-02 06:42:40,618 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-35) domain 44559ec0-6bef-491c-bbd6-e67278f37987:export in problem. vds: rack >2014-08-02 06:44:13,987 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-49) Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2014-08-02 06:44:13,992 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-49) [72c79e4e] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-08-02 06:44:13,995 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-49) [72c79e4e] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Active to Unknown, reason : null >2014-08-02 06:44:13,997 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-49) [72c79e4e] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Active to Unknown, reason : null >2014-08-02 06:44:14,011 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-49) [72c79e4e] Correlation ID: 72c79e4e, Call Stack: null, Custom Event ID: -1, Message: Invalid status on Data Center Default. Setting Data Center status to Non Responsive (On host rack, Error: Cannot access master storage domain). >2014-08-02 06:46:14,039 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-49) [72c79e4e] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2014-08-02 06:46:14,041 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-49) [72c79e4e] hostFromVds::selectedVds - rack, spmStatus returned null! >2014-08-02 06:46:14,042 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-49) [72c79e4e] IrsBroker::Failed::GetStoragePoolInfoVDS >2014-08-02 06:46:14,043 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-49) [72c79e4e] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2014-08-02 06:46:14,076 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-25) [27ab8ac7] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage >2014-08-02 06:46:14,093 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-25) [27ab8ac7] START, ResetIrsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, ignoreStopFailed = true), log id: 7e61050b >2014-08-02 06:46:14,099 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-25) [27ab8ac7] START, SpmStopVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9), log id: 10e50f4b >2014-08-02 06:46:14,105 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-25) [27ab8ac7] SpmStopVDSCommand::Stopping SPM on vds rack, pool id 00000002-0002-0002-0002-0000000001d9 >2014-08-02 06:48:14,011 WARN [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-25) [27ab8ac7] could not stop spm of pool 00000002-0002-0002-0002-0000000001d9 on vds 17f7fe56-5f32-40ef-b098-611b84484e0b - reason: org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcRunTimeException: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException >2014-08-02 06:48:14,013 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-25) [27ab8ac7] FINISH, SpmStopVDSCommand, log id: 10e50f4b >2014-08-02 06:48:14,016 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ListVDSCommand] (DefaultQuartzScheduler_Worker-88) Command ListVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-08-02 06:48:14,017 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-94) Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: VDSNetworkException: org.apache.commons.httpclient.NoHttpResponseException: The server 10.34.62.208 failed to respond >2014-08-02 06:48:14,018 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-88) Host rack is not responding. It will stay in Connecting state for a grace period of $80 seconds and after that an attempt to fence the host will be issued. >2014-08-02 06:48:14,021 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-94) hostFromVds::selectedVds - rack, spmStatus returned null! >2014-08-02 06:48:14,024 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-25) [27ab8ac7] FINISH, ResetIrsVDSCommand, log id: 7e61050b >2014-08-02 06:48:14,022 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-88) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:406) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.fetchRunningVms(VdsUpdateRunTimeInfo.java:971) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVmStats(VdsUpdateRunTimeInfo.java:920) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:638) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:235) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) [:1.7.0_55] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_55] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_55] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_55] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_55] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_55] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_55] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_55] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy82.list(Unknown Source) > at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source) [:1.7.0_55] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_55] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_55] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_55] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_55] > >2014-08-02 06:48:14,034 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-25) [27ab8ac7] Correlation ID: 27ab8ac7, Job ID: 99448b18-cda2-4769-96af-892b76b11b35, Call Stack: null, Custom Event ID: -1, Message: Failed to Reconstruct Master Domain for Data Center Default. >2014-08-02 06:48:14,053 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-88) Failed to refresh VDS , vds = 17f7fe56-5f32-40ef-b098-611b84484e0b : rack, VDS Network Error, continuing. >java.net.ConnectException: Connection refused >2014-08-02 06:48:14,057 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-9) [72c79e4e] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue >2014-08-02 06:48:14,059 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-33) starting processDomainRecovery for domain 44559ec0-6bef-491c-bbd6-e67278f37987:export >2014-08-02 06:48:14,125 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-33) Domain 44559ec0-6bef-491c-bbd6-e67278f37987:export was reported by all hosts in status UP as problematic. Moving the domain to NonOperational. >2014-08-02 06:48:14,141 INFO [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-33) [42b39b7a] Lock Acquired to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE >, sharedLocks= ] >2014-08-02 06:48:14,148 INFO [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-33) [42b39b7a] Running command: DeactivateStorageDomainCommand internal: true. Entities affected : ID: 44559ec0-6bef-491c-bbd6-e67278f37987 Type: Storage >2014-08-02 06:48:14,154 INFO [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-33) [42b39b7a] Lock freed to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE >, sharedLocks= ] >2014-08-02 06:48:14,163 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-33) [42b39b7a] Correlation ID: 42b39b7a, Job ID: f218d5ce-cd5c-46a6-869e-f4a01de4c40d, Call Stack: null, Custom Event ID: -1, Message: Storage Domain export (Data Center Default) was deactivated by system because it's not visible by any of the hosts. >2014-08-02 06:48:14,168 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-33) [42b39b7a] Removing vds [] from the domain in maintenance cache >2014-08-02 06:48:14,169 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-33) [42b39b7a] Removing host(s) [] from hosts unseen domain report cache >2014-08-02 06:48:14,171 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-13) starting processDomainRecovery for domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 >2014-08-02 06:48:14,176 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-13) Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 was reported by all hosts in status UP as problematic. Not moving the domain to NonOperational because it is being reconstructed now. >2014-08-02 06:48:14,193 WARN [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-13) [403a3975] CanDoAction of action ReconstructMasterDomain failed. Reasons:VAR__ACTION__RECONSTRUCT_MASTER,VAR__TYPE__STORAGE__DOMAIN,ACTION_TYPE_FAILED_NO_VDS_IN_POOL >2014-08-02 06:48:14,199 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-13) [403a3975] Removing vds [] from the domain in maintenance cache >2014-08-02 06:48:14,200 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-13) [403a3975] Removing host(s) [] from hosts unseen domain report cache >2014-08-02 06:48:14,201 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-9) [72c79e4e] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue >2014-08-02 06:48:17,303 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-39) [4a2f7e33] START, GetHardwareInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]), log id: 602557e9 >2014-08-02 06:48:17,360 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-39) [4a2f7e33] FINISH, GetHardwareInfoVDSCommand, log id: 602557e9 >2014-08-02 06:48:17,383 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-39) [1fef041f] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS >2014-08-02 06:48:17,397 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-39) [4ae34136] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-08-02 06:48:17,405 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-39) [39b61b80] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-08-02 06:48:17,414 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-39) [39b61b80] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, storageType = NFS, connectionList = [{ id: 05047c9a-152e-4ef2-b118-4a1753c1eae0, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35export, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: d22a875 >2014-08-02 06:48:17,477 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-39) [39b61b80] FINISH, ConnectStorageServerVDSCommand, return: {5083f370-af09-457b-9b73-3440e8b65a07=0, 05047c9a-152e-4ef2-b118-4a1753c1eae0=0}, log id: d22a875 >2014-08-02 06:48:17,479 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-39) [39b61b80] Host rack storage connection was succeeded >2014-08-02 06:48:17,486 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-7) START, ConnectStoragePoolVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, masterVersion = 1), log id: 22740ece >2014-08-02 06:48:24,027 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-92) Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Inactive to Unknown, reason : no reporting hosts >2014-08-02 06:48:24,031 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-92) Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Inactive to Unknown, reason : no reporting hosts >2014-08-02 06:49:17,914 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-7) FINISH, ConnectStoragePoolVDSCommand, log id: 22740ece >2014-08-02 06:49:17,952 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-39) [39b61b80] START, SetMOMPolicyParametersVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 18e75966 >2014-08-02 06:49:18,048 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-39) [39b61b80] FINISH, SetMOMPolicyParametersVDSCommand, log id: 18e75966 >2014-08-02 06:49:18,054 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-39) [39b61b80] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host rack was set to Up. >2014-08-02 06:49:18,061 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-39) [39b61b80] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host rack. >2014-08-02 06:49:18,083 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-39) [778da158] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS >2014-08-02 06:49:24,209 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-97) [1b0a6877] hostFromVds::selectedVds - rack, spmStatus Free, storage pool Default >2014-08-02 06:49:24,214 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-97) [1b0a6877] starting spm on vds rack, storage pool Default, prevId -1, LVER -1 >2014-08-02 06:49:24,217 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-97) [1b0a6877] START, SpmStartVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, prevId=-1, prevLVER=-1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 4afb8e4a >2014-08-02 06:49:24,265 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-97) [1b0a6877] spmStart polling started: taskId = 48f1f0a3-dfd7-41e7-a98c-0648e13a68f4 >2014-08-02 06:49:25,272 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-97) [1b0a6877] spmStart polling ended: taskId = 48f1f0a3-dfd7-41e7-a98c-0648e13a68f4 task status = finished >2014-08-02 06:49:25,327 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-97) [1b0a6877] spmStart polling ended, spm status: SPM >2014-08-02 06:49:25,330 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-97) [1b0a6877] START, HSMClearTaskVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, taskId=48f1f0a3-dfd7-41e7-a98c-0648e13a68f4), log id: 49d938e4 >2014-08-02 06:49:25,376 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-97) [1b0a6877] FINISH, HSMClearTaskVDSCommand, log id: 49d938e4 >2014-08-02 06:49:25,377 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-97) [1b0a6877] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@4f60e75e, log id: 4afb8e4a >2014-08-02 06:49:25,381 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-97) [1b0a6877] Initialize Irs proxy from vds: 10.34.62.208 >2014-08-02 06:49:25,387 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-97) [1b0a6877] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208). >2014-08-02 06:49:25,396 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-30) [1b0a6877] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 2152aa5f >2014-08-02 06:49:25,433 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-30) [1b0a6877] -- executeIrsBrokerCommand: Attempting on storage pool 00000002-0002-0002-0002-0000000001d9 >2014-08-02 06:49:25,435 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-30) [1b0a6877] START, HSMGetAllTasksInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 3b43c857 >2014-08-02 06:49:25,441 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-30) [1b0a6877] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 3b43c857 >2014-08-02 06:49:25,442 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-30) [1b0a6877] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 2152aa5f >2014-08-02 06:49:25,443 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (org.ovirt.thread.pool-8-thread-30) [1b0a6877] Discovered no tasks on Storage Pool Default >2014-08-02 07:04:10,278 WARN [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-8-thread-42) Executing a command: java.util.concurrent.FutureTask , but note that there are 1 tasks in the queue. >2014-08-02 07:23:55,592 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-7) [42c2440e] Principal admin::internal synchronized >2014-08-02 07:24:23,414 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-46) [56f1581c] Attempting to update VMs/Templates Ovf. >2014-08-02 08:23:55,597 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-77) [2e3238db] Principal admin::internal synchronized >2014-08-02 08:24:23,427 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-80) Attempting to update VMs/Templates Ovf. >2014-08-02 08:27:38,636 WARN [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-8-thread-13) Executing a command: java.util.concurrent.FutureTask , but note that there are 1 tasks in the queue. >2014-08-02 09:23:55,603 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-66) [2809936b] Principal admin::internal synchronized >2014-08-02 09:24:23,443 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-31) Attempting to update VMs/Templates Ovf. >2014-08-02 09:49:00,318 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-12) domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 in problem. vds: rack >2014-08-02 09:49:15,433 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-49) Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 recovered from problem. vds: rack >2014-08-02 09:49:15,435 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-49) Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 has recovered from problem. No active host in the DC is reporting it as problematic, so clearing the domain recovery timer. >2014-08-02 09:54:02,727 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-49) domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 in problem. vds: rack >2014-08-02 09:54:02,729 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-49) domain 44559ec0-6bef-491c-bbd6-e67278f37987:export in problem. vds: rack >2014-08-02 09:55:41,513 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-100) [52a75f18] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2014-08-02 09:55:41,517 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-100) [3b9a9fa6] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-08-02 09:55:41,521 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-100) [3b9a9fa6] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Active to Unknown, reason : null >2014-08-02 09:55:41,524 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-100) [3b9a9fa6] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Active to Unknown, reason : null >2014-08-02 09:55:41,543 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-100) [3b9a9fa6] Correlation ID: 3b9a9fa6, Call Stack: null, Custom Event ID: -1, Message: Invalid status on Data Center Default. Setting Data Center status to Non Responsive (On host rack, Error: Cannot access master storage domain). >2014-08-02 09:57:41,558 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-100) [3b9a9fa6] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2014-08-02 09:57:41,560 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-100) [3b9a9fa6] hostFromVds::selectedVds - rack, spmStatus returned null! >2014-08-02 09:57:41,561 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-100) [3b9a9fa6] IrsBroker::Failed::GetStoragePoolInfoVDS >2014-08-02 09:57:41,562 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-100) [3b9a9fa6] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2014-08-02 09:57:41,596 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-43) [1e342011] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage >2014-08-02 09:57:41,613 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-43) [1e342011] START, ResetIrsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, ignoreStopFailed = true), log id: 5bf4ad33 >2014-08-02 09:57:41,616 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-43) [1e342011] START, SpmStopVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9), log id: 3b7ebd93 >2014-08-02 09:57:41,622 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-43) [1e342011] SpmStopVDSCommand::Stopping SPM on vds rack, pool id 00000002-0002-0002-0002-0000000001d9 >2014-08-02 09:59:41,534 WARN [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-43) [1e342011] could not stop spm of pool 00000002-0002-0002-0002-0000000001d9 on vds 17f7fe56-5f32-40ef-b098-611b84484e0b - reason: org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcRunTimeException: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException >2014-08-02 09:59:41,538 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-43) [1e342011] FINISH, SpmStopVDSCommand, log id: 3b7ebd93 >2014-08-02 09:59:41,537 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ListVDSCommand] (DefaultQuartzScheduler_Worker-92) Command ListVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-08-02 09:59:41,541 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-86) [6452f878] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: VDSNetworkException: org.apache.commons.httpclient.NoHttpResponseException: The server 10.34.62.208 failed to respond >2014-08-02 09:59:41,543 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-86) [6452f878] hostFromVds::selectedVds - rack, spmStatus returned null! >2014-08-02 09:59:41,541 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-92) Host rack is not responding. It will stay in Connecting state for a grace period of $80 seconds and after that an attempt to fence the host will be issued. >2014-08-02 09:59:41,545 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-43) [1e342011] FINISH, ResetIrsVDSCommand, log id: 5bf4ad33 >2014-08-02 09:59:41,546 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-92) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:406) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.fetchRunningVms(VdsUpdateRunTimeInfo.java:971) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVmStats(VdsUpdateRunTimeInfo.java:920) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:638) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:235) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) [:1.7.0_55] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_55] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_55] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_55] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_55] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_55] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_55] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_55] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy82.list(Unknown Source) > at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source) [:1.7.0_55] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_55] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_55] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_55] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_55] > >2014-08-02 09:59:41,555 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-43) [1e342011] Correlation ID: 1e342011, Job ID: d536ca2f-e474-44f5-85fd-b25232ac2e26, Call Stack: null, Custom Event ID: -1, Message: Failed to Reconstruct Master Domain for Data Center Default. >2014-08-02 09:59:41,584 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-92) Failed to refresh VDS , vds = 17f7fe56-5f32-40ef-b098-611b84484e0b : rack, VDS Network Error, continuing. >java.net.ConnectException: Connection refused >2014-08-02 09:59:41,585 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-38) [3b9a9fa6] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue >2014-08-02 09:59:41,587 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-25) starting processDomainRecovery for domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 >2014-08-02 09:59:41,652 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-25) Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 was reported by all hosts in status UP as problematic. Not moving the domain to NonOperational because it is being reconstructed now. >2014-08-02 09:59:41,683 WARN [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-25) [36b4aedd] CanDoAction of action ReconstructMasterDomain failed. Reasons:VAR__ACTION__RECONSTRUCT_MASTER,VAR__TYPE__STORAGE__DOMAIN,ACTION_TYPE_FAILED_NO_VDS_IN_POOL >2014-08-02 09:59:41,691 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-25) [36b4aedd] Removing vds [] from the domain in maintenance cache >2014-08-02 09:59:41,692 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-25) [36b4aedd] Removing host(s) [] from hosts unseen domain report cache >2014-08-02 09:59:41,693 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-38) [3b9a9fa6] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue >2014-08-02 09:59:41,695 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-21) starting processDomainRecovery for domain 44559ec0-6bef-491c-bbd6-e67278f37987:export >2014-08-02 09:59:41,700 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-21) Domain 44559ec0-6bef-491c-bbd6-e67278f37987:export was reported by all hosts in status UP as problematic. Moving the domain to NonOperational. >2014-08-02 09:59:41,715 INFO [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-21) [3fa1a8be] Lock Acquired to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE >, sharedLocks= ] >2014-08-02 09:59:41,722 INFO [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-21) [3fa1a8be] Running command: DeactivateStorageDomainCommand internal: true. Entities affected : ID: 44559ec0-6bef-491c-bbd6-e67278f37987 Type: Storage >2014-08-02 09:59:41,728 INFO [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-21) [3fa1a8be] Lock freed to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE >, sharedLocks= ] >2014-08-02 09:59:41,737 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-21) [3fa1a8be] Correlation ID: 3fa1a8be, Job ID: 951aa6d2-965b-4f36-9026-c6075f7dffcb, Call Stack: null, Custom Event ID: -1, Message: Storage Domain export (Data Center Default) was deactivated by system because it's not visible by any of the hosts. >2014-08-02 09:59:41,742 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-21) [3fa1a8be] Removing vds [] from the domain in maintenance cache >2014-08-02 09:59:41,743 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-21) [3fa1a8be] Removing host(s) [] from hosts unseen domain report cache >2014-08-02 09:59:44,844 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-81) START, GetHardwareInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]), log id: 240a17da >2014-08-02 09:59:44,893 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-81) FINISH, GetHardwareInfoVDSCommand, log id: 240a17da >2014-08-02 09:59:44,923 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-81) [37653756] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS >2014-08-02 09:59:44,977 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-81) [5c87d5a2] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-08-02 09:59:44,985 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-81) [240bfd9c] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-08-02 09:59:45,002 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-81) [240bfd9c] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, storageType = NFS, connectionList = [{ id: 05047c9a-152e-4ef2-b118-4a1753c1eae0, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35export, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 31af56e4 >2014-08-02 09:59:45,027 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-81) [240bfd9c] FINISH, ConnectStorageServerVDSCommand, return: {5083f370-af09-457b-9b73-3440e8b65a07=0, 05047c9a-152e-4ef2-b118-4a1753c1eae0=0}, log id: 31af56e4 >2014-08-02 09:59:45,028 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-81) [240bfd9c] Host rack storage connection was succeeded >2014-08-02 09:59:45,036 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-28) START, ConnectStoragePoolVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, masterVersion = 1), log id: 4e3184fd >2014-08-02 09:59:51,548 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-26) [6bd9692] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Inactive to Unknown, reason : no reporting hosts >2014-08-02 09:59:51,554 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-26) [6bd9692] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Inactive to Unknown, reason : no reporting hosts >2014-08-02 10:00:45,446 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-28) FINISH, ConnectStoragePoolVDSCommand, log id: 4e3184fd >2014-08-02 10:00:45,459 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-81) [240bfd9c] START, SetMOMPolicyParametersVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 5f639bd9 >2014-08-02 10:00:45,553 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-81) [240bfd9c] FINISH, SetMOMPolicyParametersVDSCommand, log id: 5f639bd9 >2014-08-02 10:00:45,560 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-81) [240bfd9c] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host rack was set to Up. >2014-08-02 10:00:45,566 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-81) [240bfd9c] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host rack. >2014-08-02 10:00:45,607 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-81) [7125e62c] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS >2014-08-02 10:00:51,713 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-27) hostFromVds::selectedVds - rack, spmStatus Free, storage pool Default >2014-08-02 10:00:51,718 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-27) starting spm on vds rack, storage pool Default, prevId -1, LVER -1 >2014-08-02 10:00:51,721 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-27) START, SpmStartVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, prevId=-1, prevLVER=-1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 68224b >2014-08-02 10:00:51,771 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-27) spmStart polling started: taskId = 477dd556-7eca-44ea-a489-060fa406db93 >2014-08-02 10:00:52,778 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-27) spmStart polling ended: taskId = 477dd556-7eca-44ea-a489-060fa406db93 task status = finished >2014-08-02 10:00:52,834 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-27) spmStart polling ended, spm status: SPM >2014-08-02 10:00:52,836 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-27) START, HSMClearTaskVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, taskId=477dd556-7eca-44ea-a489-060fa406db93), log id: 85e6f69 >2014-08-02 10:00:52,881 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-27) FINISH, HSMClearTaskVDSCommand, log id: 85e6f69 >2014-08-02 10:00:52,883 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-27) FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@5fcad40a, log id: 68224b >2014-08-02 10:00:52,888 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-27) Initialize Irs proxy from vds: 10.34.62.208 >2014-08-02 10:00:52,896 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-27) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208). >2014-08-02 10:00:52,898 WARN [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-8-thread-26) Executing a command: java.util.concurrent.FutureTask , but note that there are 1 tasks in the queue. >2014-08-02 10:00:52,908 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-26) START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 2b69f398 >2014-08-02 10:00:52,941 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-26) -- executeIrsBrokerCommand: Attempting on storage pool 00000002-0002-0002-0002-0000000001d9 >2014-08-02 10:00:52,944 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-26) START, HSMGetAllTasksInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 56712782 >2014-08-02 10:00:52,950 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-26) FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 56712782 >2014-08-02 10:00:52,951 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-26) FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 2b69f398 >2014-08-02 10:00:52,953 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (org.ovirt.thread.pool-8-thread-26) Discovered no tasks on Storage Pool Default >2014-08-02 10:10:59,639 WARN [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-8-thread-23) Executing a command: java.util.concurrent.FutureTask , but note that there are 1 tasks in the queue. >2014-08-02 10:23:55,610 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-31) Principal admin::internal synchronized >2014-08-02 10:24:23,457 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-61) [58bfe349] Attempting to update VMs/Templates Ovf. >2014-08-02 11:23:55,620 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-1) Principal admin::internal synchronized >2014-08-02 11:24:23,491 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-6) [7f1e369e] Attempting to update VMs/Templates Ovf. >2014-08-02 12:23:55,630 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-83) Principal admin::internal synchronized >2014-08-02 12:24:23,504 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-55) [7526d482] Attempting to update VMs/Templates Ovf. >2014-08-02 13:00:37,098 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-40) domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 in problem. vds: rack >2014-08-02 13:00:37,100 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-40) domain 44559ec0-6bef-491c-bbd6-e67278f37987:export in problem. vds: rack >2014-08-02 13:02:08,319 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-23) [6169221c] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2014-08-02 13:02:08,324 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-23) [3cf3236] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-08-02 13:02:08,326 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-23) [3cf3236] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Active to Unknown, reason : null >2014-08-02 13:02:08,328 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-23) [3cf3236] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Active to Unknown, reason : null >2014-08-02 13:02:08,340 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-23) [3cf3236] Correlation ID: 3cf3236, Call Stack: null, Custom Event ID: -1, Message: Invalid status on Data Center Default. Setting Data Center status to Non Responsive (On host rack, Error: Cannot access master storage domain). >2014-08-02 13:04:08,372 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-23) [3cf3236] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2014-08-02 13:04:08,374 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-23) [3cf3236] hostFromVds::selectedVds - rack, spmStatus returned null! >2014-08-02 13:04:08,375 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-23) [3cf3236] IrsBroker::Failed::GetStoragePoolInfoVDS >2014-08-02 13:04:08,376 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-23) [3cf3236] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2014-08-02 13:04:08,404 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-45) [58dd099e] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage >2014-08-02 13:04:08,421 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-45) [58dd099e] START, ResetIrsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, ignoreStopFailed = true), log id: 6e5dd12f >2014-08-02 13:04:08,424 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-45) [58dd099e] START, SpmStopVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9), log id: 400c23a6 >2014-08-02 13:04:08,430 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-45) [58dd099e] SpmStopVDSCommand::Stopping SPM on vds rack, pool id 00000002-0002-0002-0002-0000000001d9 >2014-08-02 13:05:00,003 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-80) Autorecovering 1 storage domains >2014-08-02 13:05:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-80) Autorecovering storage domains id: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 >2014-08-02 13:05:00,006 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-80) [78845858] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage >2014-08-02 13:05:00,007 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-80) [78845858] ConnectDomainToStorage. Before Connect all hosts to pool. Time:8/2/14 1:05 PM >2014-08-02 13:05:00,015 INFO [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-8-thread-40) [22b3bae8] Running command: ConnectStorageToVdsCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2014-08-02 13:05:00,020 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-40) [22b3bae8] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 12c5b7a9 >2014-08-02 13:06:08,346 WARN [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-45) [58dd099e] could not stop spm of pool 00000002-0002-0002-0002-0000000001d9 on vds 17f7fe56-5f32-40ef-b098-611b84484e0b - reason: org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcRunTimeException: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException >2014-08-02 13:06:08,348 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-45) [58dd099e] FINISH, SpmStopVDSCommand, log id: 400c23a6 >2014-08-02 13:06:08,358 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-40) [22b3bae8] Command ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-08-02 13:06:08,351 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-21) [81f884c] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: VDSNetworkException: org.apache.commons.httpclient.NoHttpResponseException: The server 10.34.62.208 failed to respond >2014-08-02 13:06:08,350 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ListVDSCommand] (DefaultQuartzScheduler_Worker-11) [aeb0141] Command ListVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-08-02 13:06:08,362 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-21) [81f884c] hostFromVds::selectedVds - rack, spmStatus returned null! >2014-08-02 13:06:08,360 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-40) [22b3bae8] FINISH, ConnectStorageServerVDSCommand, log id: 12c5b7a9 >2014-08-02 13:06:08,365 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-11) [aeb0141] Host rack is not responding. It will stay in Connecting state for a grace period of $80 seconds and after that an attempt to fence the host will be issued. >2014-08-02 13:06:08,367 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-45) [58dd099e] FINISH, ResetIrsVDSCommand, log id: 6e5dd12f >2014-08-02 13:06:08,367 ERROR [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-8-thread-40) [22b3bae8] Command org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-08-02 13:06:08,368 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-11) [aeb0141] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:406) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.fetchRunningVms(VdsUpdateRunTimeInfo.java:971) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVmStats(VdsUpdateRunTimeInfo.java:920) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:638) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:235) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) [:1.7.0_55] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_55] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_55] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_55] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_55] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_55] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_55] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_55] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy82.list(Unknown Source) > at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source) [:1.7.0_55] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_55] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_55] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_55] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_55] > >2014-08-02 13:06:08,388 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-45) [58dd099e] Correlation ID: 58dd099e, Job ID: 77c660cf-ebdc-4b4d-8ba8-d36a4d23524a, Call Stack: null, Custom Event ID: -1, Message: Failed to Reconstruct Master Domain for Data Center Default. >2014-08-02 13:06:08,373 ERROR [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-8-thread-40) [22b3bae8] Transaction rolled-back for command: org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand. >2014-08-02 13:06:08,400 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-80) [78845858] ConnectDomainToStorage. After Connect all hosts to pool. Time:8/2/14 1:06 PM >2014-08-02 13:06:08,402 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-20) [3cf3236] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue >2014-08-02 13:06:08,405 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-39) starting processDomainRecovery for domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 >2014-08-02 13:06:08,408 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-11) [aeb0141] Failed to refresh VDS , vds = 17f7fe56-5f32-40ef-b098-611b84484e0b : rack, VDS Network Error, continuing. >java.net.ConnectException: Connection refused >2014-08-02 13:06:08,458 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-39) Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 was reported by all hosts in status UP as problematic. Not moving the domain to NonOperational because it is being reconstructed now. >2014-08-02 13:06:08,478 WARN [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-39) [48c6bd11] CanDoAction of action ReconstructMasterDomain failed. Reasons:VAR__ACTION__RECONSTRUCT_MASTER,VAR__TYPE__STORAGE__DOMAIN,ACTION_TYPE_FAILED_NO_VDS_IN_POOL >2014-08-02 13:06:08,485 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-39) [48c6bd11] Removing vds [] from the domain in maintenance cache >2014-08-02 13:06:08,486 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-39) [48c6bd11] Removing host(s) [] from hosts unseen domain report cache >2014-08-02 13:06:08,487 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-20) [3cf3236] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue >2014-08-02 13:06:08,489 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-5) starting processDomainRecovery for domain 44559ec0-6bef-491c-bbd6-e67278f37987:export >2014-08-02 13:06:08,493 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-5) Domain 44559ec0-6bef-491c-bbd6-e67278f37987:export was reported by all hosts in status UP as problematic. Moving the domain to NonOperational. >2014-08-02 13:06:08,507 INFO [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-5) [7851471] Lock Acquired to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE >, sharedLocks= ] >2014-08-02 13:06:08,514 INFO [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-5) [7851471] Running command: DeactivateStorageDomainCommand internal: true. Entities affected : ID: 44559ec0-6bef-491c-bbd6-e67278f37987 Type: Storage >2014-08-02 13:06:08,519 INFO [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-5) [7851471] Lock freed to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE >, sharedLocks= ] >2014-08-02 13:06:08,530 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-5) [7851471] Correlation ID: 7851471, Job ID: 2ff17a4f-38ca-43ac-b82d-f386c591489e, Call Stack: null, Custom Event ID: -1, Message: Storage Domain export (Data Center Default) was deactivated by system because it's not visible by any of the hosts. >2014-08-02 13:06:08,535 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-5) [7851471] Removing vds [] from the domain in maintenance cache >2014-08-02 13:06:08,536 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-5) [7851471] Removing host(s) [] from hosts unseen domain report cache >2014-08-02 13:06:11,653 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-48) START, GetHardwareInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]), log id: 48714368 >2014-08-02 13:06:11,702 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-48) FINISH, GetHardwareInfoVDSCommand, log id: 48714368 >2014-08-02 13:06:11,729 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-48) [46213537] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS >2014-08-02 13:06:11,742 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-48) [239799d0] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-08-02 13:06:11,749 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-48) [3ddc2a6e] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-08-02 13:06:11,755 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-48) [3ddc2a6e] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, storageType = NFS, connectionList = [{ id: 05047c9a-152e-4ef2-b118-4a1753c1eae0, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35export, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 41607b >2014-08-02 13:06:11,816 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-48) [3ddc2a6e] FINISH, ConnectStorageServerVDSCommand, return: {5083f370-af09-457b-9b73-3440e8b65a07=0, 05047c9a-152e-4ef2-b118-4a1753c1eae0=0}, log id: 41607b >2014-08-02 13:06:11,818 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-48) [3ddc2a6e] Host rack storage connection was succeeded >2014-08-02 13:06:11,824 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-18) START, ConnectStoragePoolVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, masterVersion = 1), log id: 117dd06e >2014-08-02 13:06:18,370 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-10) Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Inactive to Unknown, reason : no reporting hosts >2014-08-02 13:06:18,374 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-10) Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Inactive to Unknown, reason : no reporting hosts >2014-08-02 13:07:12,228 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-18) FINISH, ConnectStoragePoolVDSCommand, log id: 117dd06e >2014-08-02 13:07:12,240 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-48) [3ddc2a6e] START, SetMOMPolicyParametersVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 2a022410 >2014-08-02 13:07:12,333 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-48) [3ddc2a6e] FINISH, SetMOMPolicyParametersVDSCommand, log id: 2a022410 >2014-08-02 13:07:12,341 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-48) [3ddc2a6e] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host rack was set to Up. >2014-08-02 13:07:12,352 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-48) [3ddc2a6e] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host rack. >2014-08-02 13:07:12,372 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-48) [14cac44e] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS >2014-08-02 13:07:18,496 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-63) hostFromVds::selectedVds - rack, spmStatus Free, storage pool Default >2014-08-02 13:07:18,502 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-63) starting spm on vds rack, storage pool Default, prevId -1, LVER -1 >2014-08-02 13:07:18,505 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-63) START, SpmStartVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, prevId=-1, prevLVER=-1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 2384de27 >2014-08-02 13:07:18,553 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-63) spmStart polling started: taskId = eea8bd7b-857f-40af-92f5-21f814dc073a >2014-08-02 13:07:19,559 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-63) spmStart polling ended: taskId = eea8bd7b-857f-40af-92f5-21f814dc073a task status = finished >2014-08-02 13:07:19,615 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-63) spmStart polling ended, spm status: SPM >2014-08-02 13:07:19,618 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-63) START, HSMClearTaskVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, taskId=eea8bd7b-857f-40af-92f5-21f814dc073a), log id: 65abbc00 >2014-08-02 13:07:19,663 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-63) FINISH, HSMClearTaskVDSCommand, log id: 65abbc00 >2014-08-02 13:07:19,664 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-63) FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@559e77b1, log id: 2384de27 >2014-08-02 13:07:19,669 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-63) Initialize Irs proxy from vds: 10.34.62.208 >2014-08-02 13:07:19,676 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-63) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208). >2014-08-02 13:07:19,690 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-30) START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 27d5af23 >2014-08-02 13:07:19,721 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-30) -- executeIrsBrokerCommand: Attempting on storage pool 00000002-0002-0002-0002-0000000001d9 >2014-08-02 13:07:19,724 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-30) START, HSMGetAllTasksInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 4122154e >2014-08-02 13:07:19,729 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-30) FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 4122154e >2014-08-02 13:07:19,730 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-30) FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 27d5af23 >2014-08-02 13:07:19,731 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (org.ovirt.thread.pool-8-thread-30) Discovered no tasks on Storage Pool Default >2014-08-02 13:23:55,641 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-78) Principal admin::internal synchronized >2014-08-02 13:24:23,520 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-18) [5497a121] Attempting to update VMs/Templates Ovf. >2014-08-02 14:23:55,648 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-91) [64566e87] Principal admin::internal synchronized >2014-08-02 14:24:23,555 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-54) [14ab6114] Attempting to update VMs/Templates Ovf. >2014-08-02 15:23:55,655 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-61) [58bfe349] Principal admin::internal synchronized >2014-08-02 15:24:23,566 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-29) [327b509d] Attempting to update VMs/Templates Ovf. >2014-08-02 16:06:55,210 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-42) domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 in problem. vds: rack >2014-08-02 16:07:10,328 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-5) Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 recovered from problem. vds: rack >2014-08-02 16:07:10,329 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-5) Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 has recovered from problem. No active host in the DC is reporting it as problematic, so clearing the domain recovery timer. >2014-08-02 16:12:12,933 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-9) domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 in problem. vds: rack >2014-08-02 16:12:12,935 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-9) domain 44559ec0-6bef-491c-bbd6-e67278f37987:export in problem. vds: rack >2014-08-02 16:13:38,849 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-35) [78e7d47b] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2014-08-02 16:13:38,854 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-35) [5b4b3aec] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-08-02 16:13:38,857 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-35) [5b4b3aec] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Active to Unknown, reason : null >2014-08-02 16:13:38,859 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-35) [5b4b3aec] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Active to Unknown, reason : null >2014-08-02 16:13:38,870 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-35) [5b4b3aec] Correlation ID: 5b4b3aec, Call Stack: null, Custom Event ID: -1, Message: Invalid status on Data Center Default. Setting Data Center status to Non Responsive (On host rack, Error: Cannot access master storage domain). >2014-08-02 16:15:38,875 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-35) [5b4b3aec] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2014-08-02 16:15:38,877 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-35) [5b4b3aec] hostFromVds::selectedVds - rack, spmStatus returned null! >2014-08-02 16:15:38,878 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-35) [5b4b3aec] IrsBroker::Failed::GetStoragePoolInfoVDS >2014-08-02 16:15:38,879 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-35) [5b4b3aec] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2014-08-02 16:15:38,918 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-34) [69a4c1a7] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage >2014-08-02 16:15:38,937 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-34) [69a4c1a7] START, ResetIrsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, ignoreStopFailed = true), log id: 7fc31ce5 >2014-08-02 16:15:38,941 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-34) [69a4c1a7] START, SpmStopVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9), log id: 25df98b5 >2014-08-02 16:15:38,946 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-34) [69a4c1a7] SpmStopVDSCommand::Stopping SPM on vds rack, pool id 00000002-0002-0002-0002-0000000001d9 >2014-08-02 16:17:38,848 WARN [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-34) [69a4c1a7] could not stop spm of pool 00000002-0002-0002-0002-0000000001d9 on vds 17f7fe56-5f32-40ef-b098-611b84484e0b - reason: org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcRunTimeException: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException >2014-08-02 16:17:38,849 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-95) [4c7ac6a1] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: VDSNetworkException: org.apache.commons.httpclient.NoHttpResponseException: The server 10.34.62.208 failed to respond >2014-08-02 16:17:38,849 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ListVDSCommand] (DefaultQuartzScheduler_Worker-94) Command ListVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-08-02 16:17:38,853 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-95) [4c7ac6a1] hostFromVds::selectedVds - rack, spmStatus returned null! >2014-08-02 16:17:38,851 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-34) [69a4c1a7] FINISH, SpmStopVDSCommand, log id: 25df98b5 >2014-08-02 16:17:38,855 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-94) Host rack is not responding. It will stay in Connecting state for a grace period of $80 seconds and after that an attempt to fence the host will be issued. >2014-08-02 16:17:38,858 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-94) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:406) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.fetchRunningVms(VdsUpdateRunTimeInfo.java:971) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVmStats(VdsUpdateRunTimeInfo.java:920) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:638) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:235) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) [:1.7.0_55] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_55] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_55] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_55] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_55] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_55] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_55] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_55] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy82.list(Unknown Source) > at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source) [:1.7.0_55] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_55] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_55] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_55] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_55] > >2014-08-02 16:17:38,867 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-34) [69a4c1a7] FINISH, ResetIrsVDSCommand, log id: 7fc31ce5 >2014-08-02 16:17:38,891 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-94) Failed to refresh VDS , vds = 17f7fe56-5f32-40ef-b098-611b84484e0b : rack, VDS Network Error, continuing. >java.net.ConnectException: Connection refused >2014-08-02 16:17:38,896 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-34) [69a4c1a7] Correlation ID: 69a4c1a7, Job ID: becb973f-0f07-4edb-a6b4-483fa838249a, Call Stack: null, Custom Event ID: -1, Message: Failed to Reconstruct Master Domain for Data Center Default. >2014-08-02 16:17:38,902 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-45) [5b4b3aec] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue >2014-08-02 16:17:38,904 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-16) starting processDomainRecovery for domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 >2014-08-02 16:17:38,956 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-16) Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 was reported by all hosts in status UP as problematic. Not moving the domain to NonOperational because it is being reconstructed now. >2014-08-02 16:17:38,982 WARN [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-16) [17131d67] CanDoAction of action ReconstructMasterDomain failed. Reasons:VAR__ACTION__RECONSTRUCT_MASTER,VAR__TYPE__STORAGE__DOMAIN,ACTION_TYPE_FAILED_NO_VDS_IN_POOL >2014-08-02 16:17:38,989 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-16) [17131d67] Removing vds [] from the domain in maintenance cache >2014-08-02 16:17:38,990 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-16) [17131d67] Removing host(s) [] from hosts unseen domain report cache >2014-08-02 16:17:38,991 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-45) [5b4b3aec] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue >2014-08-02 16:17:38,993 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-1) starting processDomainRecovery for domain 44559ec0-6bef-491c-bbd6-e67278f37987:export >2014-08-02 16:17:38,997 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-1) Domain 44559ec0-6bef-491c-bbd6-e67278f37987:export was reported by all hosts in status UP as problematic. Moving the domain to NonOperational. >2014-08-02 16:17:39,011 INFO [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-1) [d22051a] Lock Acquired to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE >, sharedLocks= ] >2014-08-02 16:17:39,017 INFO [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-1) [d22051a] Running command: DeactivateStorageDomainCommand internal: true. Entities affected : ID: 44559ec0-6bef-491c-bbd6-e67278f37987 Type: Storage >2014-08-02 16:17:39,023 INFO [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-1) [d22051a] Lock freed to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE >, sharedLocks= ] >2014-08-02 16:17:39,031 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-1) [d22051a] Correlation ID: d22051a, Job ID: 43c67060-04e2-4496-bffc-59a9c70140cf, Call Stack: null, Custom Event ID: -1, Message: Storage Domain export (Data Center Default) was deactivated by system because it's not visible by any of the hosts. >2014-08-02 16:17:39,037 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-1) [d22051a] Removing vds [] from the domain in maintenance cache >2014-08-02 16:17:39,038 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-1) [d22051a] Removing host(s) [] from hosts unseen domain report cache >2014-08-02 16:17:42,137 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-70) START, GetHardwareInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]), log id: 435b5802 >2014-08-02 16:17:42,190 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-70) FINISH, GetHardwareInfoVDSCommand, log id: 435b5802 >2014-08-02 16:17:42,231 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-70) [5d34f3b] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS >2014-08-02 16:17:42,241 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-70) [7b012582] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-08-02 16:17:42,251 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-70) [7131f834] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-08-02 16:17:42,257 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-70) [7131f834] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, storageType = NFS, connectionList = [{ id: 05047c9a-152e-4ef2-b118-4a1753c1eae0, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35export, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 420fb9ad >2014-08-02 16:17:42,316 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-70) [7131f834] FINISH, ConnectStorageServerVDSCommand, return: {5083f370-af09-457b-9b73-3440e8b65a07=0, 05047c9a-152e-4ef2-b118-4a1753c1eae0=0}, log id: 420fb9ad >2014-08-02 16:17:42,317 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-70) [7131f834] Host rack storage connection was succeeded >2014-08-02 16:17:42,324 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-39) START, ConnectStoragePoolVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, masterVersion = 1), log id: 51df7ecf >2014-08-02 16:17:48,860 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-17) Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Inactive to Unknown, reason : no reporting hosts >2014-08-02 16:17:48,866 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-17) Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Inactive to Unknown, reason : no reporting hosts >2014-08-02 16:18:42,714 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-39) FINISH, ConnectStoragePoolVDSCommand, log id: 51df7ecf >2014-08-02 16:18:42,728 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-70) [7131f834] START, SetMOMPolicyParametersVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 1baf9cdb >2014-08-02 16:18:42,821 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-70) [7131f834] FINISH, SetMOMPolicyParametersVDSCommand, log id: 1baf9cdb >2014-08-02 16:18:42,828 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-70) [7131f834] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host rack was set to Up. >2014-08-02 16:18:42,833 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-70) [7131f834] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host rack. >2014-08-02 16:18:42,849 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-70) [593e74a7] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS >2014-08-02 16:18:48,985 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-3) hostFromVds::selectedVds - rack, spmStatus Free, storage pool Default >2014-08-02 16:18:48,989 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-3) starting spm on vds rack, storage pool Default, prevId -1, LVER -1 >2014-08-02 16:18:48,991 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-3) START, SpmStartVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, prevId=-1, prevLVER=-1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 65f4d717 >2014-08-02 16:18:49,039 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-3) spmStart polling started: taskId = eded74c1-eabd-4e84-a1c6-43f09092abb4 >2014-08-02 16:18:50,046 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-3) spmStart polling ended: taskId = eded74c1-eabd-4e84-a1c6-43f09092abb4 task status = finished >2014-08-02 16:18:50,101 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-3) spmStart polling ended, spm status: SPM >2014-08-02 16:18:50,103 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-3) START, HSMClearTaskVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, taskId=eded74c1-eabd-4e84-a1c6-43f09092abb4), log id: 4b4ea4c3 >2014-08-02 16:18:50,148 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-3) FINISH, HSMClearTaskVDSCommand, log id: 4b4ea4c3 >2014-08-02 16:18:50,149 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-3) FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@59095303, log id: 65f4d717 >2014-08-02 16:18:50,153 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-3) Initialize Irs proxy from vds: 10.34.62.208 >2014-08-02 16:18:50,159 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-3) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208). >2014-08-02 16:18:50,167 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-46) START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 23fc4bab >2014-08-02 16:18:50,205 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-46) -- executeIrsBrokerCommand: Attempting on storage pool 00000002-0002-0002-0002-0000000001d9 >2014-08-02 16:18:50,206 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-46) START, HSMGetAllTasksInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 190252c0 >2014-08-02 16:18:50,212 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-46) FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 190252c0 >2014-08-02 16:18:50,213 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-46) FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 23fc4bab >2014-08-02 16:18:50,214 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (org.ovirt.thread.pool-8-thread-46) Discovered no tasks on Storage Pool Default >2014-08-02 16:23:55,663 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-71) [6cd4f56d] Principal admin::internal synchronized >2014-08-02 16:24:23,581 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-94) Attempting to update VMs/Templates Ovf. >2014-08-02 17:23:55,673 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-93) [156fe0ca] Principal admin::internal synchronized >2014-08-02 17:24:23,600 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-15) [39225a1e] Attempting to update VMs/Templates Ovf. >2014-08-02 18:23:55,682 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-2) Principal admin::internal synchronized >2014-08-02 18:24:23,639 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-83) Attempting to update VMs/Templates Ovf. >2014-08-02 19:18:30,981 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-6) domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 in problem. vds: rack >2014-08-02 19:18:30,983 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-6) domain 44559ec0-6bef-491c-bbd6-e67278f37987:export in problem. vds: rack >2014-08-02 19:20:05,871 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-45) [18f9dc27] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2014-08-02 19:20:05,876 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-45) [22578e65] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-08-02 19:20:05,879 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-45) [22578e65] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Active to Unknown, reason : null >2014-08-02 19:20:05,881 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-45) [22578e65] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Active to Unknown, reason : null >2014-08-02 19:20:05,892 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-45) [22578e65] Correlation ID: 22578e65, Call Stack: null, Custom Event ID: -1, Message: Invalid status on Data Center Default. Setting Data Center status to Non Responsive (On host rack, Error: Cannot access master storage domain). >2014-08-02 19:22:05,944 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-45) [22578e65] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2014-08-02 19:22:05,946 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-45) [22578e65] hostFromVds::selectedVds - rack, spmStatus returned null! >2014-08-02 19:22:05,947 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-45) [22578e65] IrsBroker::Failed::GetStoragePoolInfoVDS >2014-08-02 19:22:05,948 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-45) [22578e65] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2014-08-02 19:22:05,987 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-26) [3dec9b1a] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage >2014-08-02 19:22:06,006 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-26) [3dec9b1a] START, ResetIrsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, ignoreStopFailed = true), log id: 35957991 >2014-08-02 19:22:06,010 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-26) [3dec9b1a] START, SpmStopVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9), log id: 32514355 >2014-08-02 19:22:06,016 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-26) [3dec9b1a] SpmStopVDSCommand::Stopping SPM on vds rack, pool id 00000002-0002-0002-0002-0000000001d9 >2014-08-02 19:23:55,688 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-90) Principal admin::internal synchronized >2014-08-02 19:24:05,915 WARN [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-26) [3dec9b1a] could not stop spm of pool 00000002-0002-0002-0002-0000000001d9 on vds 17f7fe56-5f32-40ef-b098-611b84484e0b - reason: org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcRunTimeException: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException >2014-08-02 19:24:05,920 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-26) [3dec9b1a] FINISH, SpmStopVDSCommand, log id: 32514355 >2014-08-02 19:24:05,918 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetStatsVDSCommand] (DefaultQuartzScheduler_Worker-97) [1b0a6877] Command GetStatsVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-08-02 19:24:05,926 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-1) Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: VDSNetworkException: org.apache.commons.httpclient.NoHttpResponseException: The server 10.34.62.208 failed to respond >2014-08-02 19:24:05,930 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-1) hostFromVds::selectedVds - rack, spmStatus returned null! >2014-08-02 19:24:05,928 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-97) [1b0a6877] Host rack is not responding. It will stay in Connecting state for a grace period of $80 seconds and after that an attempt to fence the host will be issued. >2014-08-02 19:24:05,928 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-26) [3dec9b1a] FINISH, ResetIrsVDSCommand, log id: 35957991 >2014-08-02 19:24:05,933 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-97) [1b0a6877] vds::refreshVdsStats Failed getVdsStats, vds = 17f7fe56-5f32-40ef-b098-611b84484e0b : rack, error = VDSNetworkException: java.net.ConnectException: Connection refused >2014-08-02 19:24:05,935 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-97) [1b0a6877] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsStats(VdsUpdateRunTimeInfo.java:669) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:623) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:235) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) [:1.7.0_55] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_55] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_55] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_55] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_55] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_55] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_55] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_55] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy82.getVdsStats(Unknown Source) > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_55] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_55] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_55] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_55] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_55] > >2014-08-02 19:24:05,948 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-26) [3dec9b1a] Correlation ID: 3dec9b1a, Job ID: a4178e58-b69f-40cd-be52-5765c6b35483, Call Stack: null, Custom Event ID: -1, Message: Failed to Reconstruct Master Domain for Data Center Default. >2014-08-02 19:24:05,976 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-2) [22578e65] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue >2014-08-02 19:24:05,978 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-22) starting processDomainRecovery for domain 44559ec0-6bef-491c-bbd6-e67278f37987:export >2014-08-02 19:24:05,977 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-97) [1b0a6877] Failed to refresh VDS , vds = 17f7fe56-5f32-40ef-b098-611b84484e0b : rack, VDS Network Error, continuing. >java.net.ConnectException: Connection refused >2014-08-02 19:24:06,032 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-22) Domain 44559ec0-6bef-491c-bbd6-e67278f37987:export was reported by all hosts in status UP as problematic. Moving the domain to NonOperational. >2014-08-02 19:24:06,048 INFO [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-22) [a7c3897] Lock Acquired to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE >, sharedLocks= ] >2014-08-02 19:24:06,054 INFO [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-22) [a7c3897] Running command: DeactivateStorageDomainCommand internal: true. Entities affected : ID: 44559ec0-6bef-491c-bbd6-e67278f37987 Type: Storage >2014-08-02 19:24:06,060 INFO [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-22) [a7c3897] Lock freed to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE >, sharedLocks= ] >2014-08-02 19:24:06,068 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-22) [a7c3897] Correlation ID: a7c3897, Job ID: 223dabc7-87ac-4af6-a6d3-a1982e8b6fcb, Call Stack: null, Custom Event ID: -1, Message: Storage Domain export (Data Center Default) was deactivated by system because it's not visible by any of the hosts. >2014-08-02 19:24:06,074 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-22) [a7c3897] Removing vds [] from the domain in maintenance cache >2014-08-02 19:24:06,075 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-22) [a7c3897] Removing host(s) [] from hosts unseen domain report cache >2014-08-02 19:24:06,077 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-8) starting processDomainRecovery for domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 >2014-08-02 19:24:06,081 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-8) Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 was reported by all hosts in status UP as problematic. Not moving the domain to NonOperational because it is being reconstructed now. >2014-08-02 19:24:06,102 WARN [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-8) [8828708] CanDoAction of action ReconstructMasterDomain failed. Reasons:VAR__ACTION__RECONSTRUCT_MASTER,VAR__TYPE__STORAGE__DOMAIN,ACTION_TYPE_FAILED_NO_VDS_IN_POOL >2014-08-02 19:24:06,109 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-8) [8828708] Removing vds [] from the domain in maintenance cache >2014-08-02 19:24:06,110 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-8) [8828708] Removing host(s) [] from hosts unseen domain report cache >2014-08-02 19:24:06,111 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-2) [22578e65] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue >2014-08-02 19:24:09,233 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-23) [3cf3236] START, GetHardwareInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]), log id: 38c6a84c >2014-08-02 19:24:09,285 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-23) [3cf3236] FINISH, GetHardwareInfoVDSCommand, log id: 38c6a84c >2014-08-02 19:24:09,310 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-23) [28763d45] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS >2014-08-02 19:24:09,360 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-23) [f3cb3e] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-08-02 19:24:09,368 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-23) [2bee177c] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-08-02 19:24:09,381 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-23) [2bee177c] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, storageType = NFS, connectionList = [{ id: 05047c9a-152e-4ef2-b118-4a1753c1eae0, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35export, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 24fbafd6 >2014-08-02 19:24:09,442 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-23) [2bee177c] FINISH, ConnectStorageServerVDSCommand, return: {5083f370-af09-457b-9b73-3440e8b65a07=0, 05047c9a-152e-4ef2-b118-4a1753c1eae0=0}, log id: 24fbafd6 >2014-08-02 19:24:09,443 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-23) [2bee177c] Host rack storage connection was succeeded >2014-08-02 19:24:09,450 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-44) START, ConnectStoragePoolVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, masterVersion = 1), log id: 3889c774 >2014-08-02 19:24:15,935 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-50) [375fcc6] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Inactive to Unknown, reason : no reporting hosts >2014-08-02 19:24:15,939 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-50) [375fcc6] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Inactive to Unknown, reason : no reporting hosts >2014-08-02 19:24:23,653 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-35) [5b4b3aec] Attempting to update VMs/Templates Ovf. >2014-08-02 19:25:09,900 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-44) FINISH, ConnectStoragePoolVDSCommand, log id: 3889c774 >2014-08-02 19:25:09,911 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-23) [2bee177c] START, SetMOMPolicyParametersVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 4d5119a3 >2014-08-02 19:25:10,003 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-23) [2bee177c] FINISH, SetMOMPolicyParametersVDSCommand, log id: 4d5119a3 >2014-08-02 19:25:10,014 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-23) [2bee177c] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host rack was set to Up. >2014-08-02 19:25:10,019 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-23) [2bee177c] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host rack. >2014-08-02 19:25:10,043 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-23) [1802a60f] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS >2014-08-02 19:25:16,047 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-9) hostFromVds::selectedVds - rack, spmStatus Free, storage pool Default >2014-08-02 19:25:16,052 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-9) starting spm on vds rack, storage pool Default, prevId -1, LVER -1 >2014-08-02 19:25:16,053 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-9) START, SpmStartVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, prevId=-1, prevLVER=-1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 394c868 >2014-08-02 19:25:16,101 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-9) spmStart polling started: taskId = 86badb6f-afcd-4701-bfc8-aed1cd304126 >2014-08-02 19:25:17,109 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-9) spmStart polling ended: taskId = 86badb6f-afcd-4701-bfc8-aed1cd304126 task status = finished >2014-08-02 19:25:17,165 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-9) spmStart polling ended, spm status: SPM >2014-08-02 19:25:17,167 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-9) START, HSMClearTaskVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, taskId=86badb6f-afcd-4701-bfc8-aed1cd304126), log id: 279944b4 >2014-08-02 19:25:17,211 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-9) FINISH, HSMClearTaskVDSCommand, log id: 279944b4 >2014-08-02 19:25:17,212 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-9) FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@1bcb2b28, log id: 394c868 >2014-08-02 19:25:17,216 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-9) Initialize Irs proxy from vds: 10.34.62.208 >2014-08-02 19:25:17,222 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-9) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208). >2014-08-02 19:25:17,224 WARN [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-8-thread-11) Executing a command: java.util.concurrent.FutureTask , but note that there are 1 tasks in the queue. >2014-08-02 19:25:17,235 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-11) START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 40042922 >2014-08-02 19:25:17,261 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-11) -- executeIrsBrokerCommand: Attempting on storage pool 00000002-0002-0002-0002-0000000001d9 >2014-08-02 19:25:17,263 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-11) START, HSMGetAllTasksInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 163b9d04 >2014-08-02 19:25:17,268 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-11) FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 163b9d04 >2014-08-02 19:25:17,269 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-11) FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 40042922 >2014-08-02 19:25:17,270 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (org.ovirt.thread.pool-8-thread-11) Discovered no tasks on Storage Pool Default >2014-08-02 20:23:55,694 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-36) [3e2ffa8] Principal admin::internal synchronized >2014-08-02 20:24:23,656 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-38) [1b9ef0f] Attempting to update VMs/Templates Ovf. >2014-08-02 21:23:55,699 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-73) Principal admin::internal synchronized >2014-08-02 21:24:23,670 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-78) Attempting to update VMs/Templates Ovf. >2014-08-02 22:23:55,707 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-60) [1cb6c42c] Principal admin::internal synchronized >2014-08-02 22:24:23,693 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-63) Attempting to update VMs/Templates Ovf. >2014-08-02 22:25:04,483 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-43) domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 in problem. vds: rack >2014-08-02 22:25:19,603 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-23) domain 44559ec0-6bef-491c-bbd6-e67278f37987:export in problem. vds: rack >2014-08-02 22:26:43,848 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-50) [375fcc6] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2014-08-02 22:26:43,855 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-50) [16080eb6] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-08-02 22:26:43,859 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-50) [16080eb6] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Active to Unknown, reason : null >2014-08-02 22:26:43,861 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-50) [16080eb6] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Active to Unknown, reason : null >2014-08-02 22:26:43,875 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-50) [16080eb6] Correlation ID: 16080eb6, Call Stack: null, Custom Event ID: -1, Message: Invalid status on Data Center Default. Setting Data Center status to Non Responsive (On host rack, Error: Cannot access master storage domain). >2014-08-02 22:28:43,900 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-50) [16080eb6] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2014-08-02 22:28:43,902 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-50) [16080eb6] hostFromVds::selectedVds - rack, spmStatus returned null! >2014-08-02 22:28:43,903 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-50) [16080eb6] IrsBroker::Failed::GetStoragePoolInfoVDS >2014-08-02 22:28:43,904 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-50) [16080eb6] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2014-08-02 22:28:43,949 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-35) [4be6f2c6] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage >2014-08-02 22:28:43,981 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-35) [4be6f2c6] START, ResetIrsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, ignoreStopFailed = true), log id: 63d0b4df >2014-08-02 22:28:43,984 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-35) [4be6f2c6] START, SpmStopVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9), log id: 3e2ae127 >2014-08-02 22:28:43,991 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-35) [4be6f2c6] SpmStopVDSCommand::Stopping SPM on vds rack, pool id 00000002-0002-0002-0002-0000000001d9 >2014-08-02 22:30:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-52) Autorecovering 1 storage domains >2014-08-02 22:30:00,005 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-52) Autorecovering storage domains id: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 >2014-08-02 22:30:00,007 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-52) [2ccfb0f3] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage >2014-08-02 22:30:00,009 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-52) [2ccfb0f3] ConnectDomainToStorage. Before Connect all hosts to pool. Time:8/2/14 10:30 PM >2014-08-02 22:30:00,018 INFO [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-8-thread-11) [27896ed2] Running command: ConnectStorageToVdsCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2014-08-02 22:30:00,027 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-11) [27896ed2] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 6544c2ce >2014-08-02 22:30:43,871 WARN [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-35) [4be6f2c6] could not stop spm of pool 00000002-0002-0002-0002-0000000001d9 on vds 17f7fe56-5f32-40ef-b098-611b84484e0b - reason: org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcRunTimeException: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException >2014-08-02 22:30:43,874 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-35) [4be6f2c6] FINISH, SpmStopVDSCommand, log id: 3e2ae127 >2014-08-02 22:30:43,874 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-11) [27896ed2] Command ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-08-02 22:30:43,872 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ListVDSCommand] (DefaultQuartzScheduler_Worker-28) [8ef1638] Command ListVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-08-02 22:30:43,880 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-28) [8ef1638] Host rack is not responding. It will stay in Connecting state for a grace period of $80 seconds and after that an attempt to fence the host will be issued. >2014-08-02 22:30:43,879 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-11) [27896ed2] FINISH, ConnectStorageServerVDSCommand, log id: 6544c2ce >2014-08-02 22:30:43,875 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-88) Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: VDSNetworkException: org.apache.commons.httpclient.NoHttpResponseException: The server 10.34.62.208 failed to respond >2014-08-02 22:30:43,884 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-88) hostFromVds::selectedVds - rack, spmStatus returned null! >2014-08-02 22:30:43,884 ERROR [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-8-thread-11) [27896ed2] Command org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-08-02 22:30:43,882 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-28) [8ef1638] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:406) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.fetchRunningVms(VdsUpdateRunTimeInfo.java:971) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVmStats(VdsUpdateRunTimeInfo.java:920) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:638) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:235) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) [:1.7.0_55] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_55] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_55] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_55] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_55] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_55] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_55] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_55] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy82.list(Unknown Source) > at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source) [:1.7.0_55] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_55] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_55] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_55] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_55] > >2014-08-02 22:30:43,881 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-35) [4be6f2c6] FINISH, ResetIrsVDSCommand, log id: 63d0b4df >2014-08-02 22:30:43,896 ERROR [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-8-thread-11) [27896ed2] Transaction rolled-back for command: org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand. >2014-08-02 22:30:43,914 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-52) [2ccfb0f3] ConnectDomainToStorage. After Connect all hosts to pool. Time:8/2/14 10:30 PM >2014-08-02 22:30:43,920 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-28) [8ef1638] Failed to refresh VDS , vds = 17f7fe56-5f32-40ef-b098-611b84484e0b : rack, VDS Network Error, continuing. >java.net.ConnectException: Connection refused >2014-08-02 22:30:43,927 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-35) [4be6f2c6] Correlation ID: 4be6f2c6, Job ID: 181aefb7-a979-458f-8d7f-96e080fe365b, Call Stack: null, Custom Event ID: -1, Message: Failed to Reconstruct Master Domain for Data Center Default. >2014-08-02 22:30:43,935 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-1) [16080eb6] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue >2014-08-02 22:30:43,937 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-46) starting processDomainRecovery for domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 >2014-08-02 22:30:44,020 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-46) Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 was reported by all hosts in status UP as problematic. Not moving the domain to NonOperational because it is being reconstructed now. >2014-08-02 22:30:44,062 WARN [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-46) [398c64db] CanDoAction of action ReconstructMasterDomain failed. Reasons:VAR__ACTION__RECONSTRUCT_MASTER,VAR__TYPE__STORAGE__DOMAIN,ACTION_TYPE_FAILED_NO_VDS_IN_POOL >2014-08-02 22:30:44,070 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-46) [398c64db] Removing vds [] from the domain in maintenance cache >2014-08-02 22:30:44,071 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-46) [398c64db] Removing host(s) [] from hosts unseen domain report cache >2014-08-02 22:30:44,072 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-1) [16080eb6] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue >2014-08-02 22:30:44,074 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-14) starting processDomainRecovery for domain 44559ec0-6bef-491c-bbd6-e67278f37987:export >2014-08-02 22:30:44,079 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-14) Domain 44559ec0-6bef-491c-bbd6-e67278f37987:export was reported by all hosts in status UP as problematic. Moving the domain to NonOperational. >2014-08-02 22:30:44,095 INFO [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-14) [2243644b] Lock Acquired to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE >, sharedLocks= ] >2014-08-02 22:30:44,102 INFO [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-14) [2243644b] Running command: DeactivateStorageDomainCommand internal: true. Entities affected : ID: 44559ec0-6bef-491c-bbd6-e67278f37987 Type: Storage >2014-08-02 22:30:44,107 INFO [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-14) [2243644b] Lock freed to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE >, sharedLocks= ] >2014-08-02 22:30:44,131 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-14) [2243644b] Correlation ID: 2243644b, Job ID: 62ef5044-ff5a-4bad-b2fc-61d7dbfb3fd2, Call Stack: null, Custom Event ID: -1, Message: Storage Domain export (Data Center Default) was deactivated by system because it's not visible by any of the hosts. >2014-08-02 22:30:44,138 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-14) [2243644b] Removing vds [] from the domain in maintenance cache >2014-08-02 22:30:44,140 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-14) [2243644b] Removing host(s) [] from hosts unseen domain report cache >2014-08-02 22:30:47,197 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-84) [7ef2d911] START, GetHardwareInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]), log id: 4c2dc6 >2014-08-02 22:30:47,247 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-84) [7ef2d911] FINISH, GetHardwareInfoVDSCommand, log id: 4c2dc6 >2014-08-02 22:30:47,275 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-84) [6eca077a] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS >2014-08-02 22:30:47,324 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-84) [2bbfe85] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-08-02 22:30:47,332 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-84) [414b367e] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-08-02 22:30:47,342 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-84) [414b367e] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, storageType = NFS, connectionList = [{ id: 05047c9a-152e-4ef2-b118-4a1753c1eae0, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35export, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 6ded47ec >2014-08-02 22:30:47,403 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-84) [414b367e] FINISH, ConnectStorageServerVDSCommand, return: {5083f370-af09-457b-9b73-3440e8b65a07=0, 05047c9a-152e-4ef2-b118-4a1753c1eae0=0}, log id: 6ded47ec >2014-08-02 22:30:47,405 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-84) [414b367e] Host rack storage connection was succeeded >2014-08-02 22:30:47,413 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-41) START, ConnectStoragePoolVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, masterVersion = 1), log id: d791798 >2014-08-02 22:30:53,888 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-53) Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Inactive to Unknown, reason : no reporting hosts >2014-08-02 22:30:53,892 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-53) Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Inactive to Unknown, reason : no reporting hosts >2014-08-02 22:31:47,815 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-41) FINISH, ConnectStoragePoolVDSCommand, log id: d791798 >2014-08-02 22:31:47,827 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-84) [414b367e] START, SetMOMPolicyParametersVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 7b057c2c >2014-08-02 22:31:47,922 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-84) [414b367e] FINISH, SetMOMPolicyParametersVDSCommand, log id: 7b057c2c >2014-08-02 22:31:47,933 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-84) [414b367e] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host rack was set to Up. >2014-08-02 22:31:47,942 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-84) [414b367e] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host rack. >2014-08-02 22:31:47,962 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-84) [59dec259] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS >2014-08-02 22:31:54,051 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-16) hostFromVds::selectedVds - rack, spmStatus Free, storage pool Default >2014-08-02 22:31:54,060 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-16) starting spm on vds rack, storage pool Default, prevId -1, LVER -1 >2014-08-02 22:31:54,067 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-16) START, SpmStartVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, prevId=-1, prevLVER=-1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 5d4de280 >2014-08-02 22:31:54,118 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-16) spmStart polling started: taskId = 6ba464a7-bab7-470e-941b-7eb78762f619 >2014-08-02 22:31:55,124 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-16) spmStart polling ended: taskId = 6ba464a7-bab7-470e-941b-7eb78762f619 task status = finished >2014-08-02 22:31:55,180 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-16) spmStart polling ended, spm status: SPM >2014-08-02 22:31:55,183 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-16) START, HSMClearTaskVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, taskId=6ba464a7-bab7-470e-941b-7eb78762f619), log id: 10bf3e13 >2014-08-02 22:31:55,228 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-16) FINISH, HSMClearTaskVDSCommand, log id: 10bf3e13 >2014-08-02 22:31:55,229 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-16) FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@6f666b43, log id: 5d4de280 >2014-08-02 22:31:55,233 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-16) Initialize Irs proxy from vds: 10.34.62.208 >2014-08-02 22:31:55,239 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-16) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208). >2014-08-02 22:31:55,247 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-10) START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: f1c5ade >2014-08-02 22:31:55,283 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-10) -- executeIrsBrokerCommand: Attempting on storage pool 00000002-0002-0002-0002-0000000001d9 >2014-08-02 22:31:55,284 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-10) START, HSMGetAllTasksInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 7bfa6bec >2014-08-02 22:31:55,290 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-10) FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 7bfa6bec >2014-08-02 22:31:55,291 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-10) FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: f1c5ade >2014-08-02 22:31:55,292 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (org.ovirt.thread.pool-8-thread-10) Discovered no tasks on Storage Pool Default >2014-08-02 23:23:55,713 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-95) [4c7ac6a1] Principal admin::internal synchronized >2014-08-02 23:24:23,730 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-85) [5e890dae] Attempting to update VMs/Templates Ovf. >2014-08-03 00:23:55,723 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-20) [367a2664] Principal admin::internal synchronized >2014-08-03 00:24:23,746 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-53) Attempting to update VMs/Templates Ovf. >2014-08-03 01:23:55,743 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-47) Principal admin::internal synchronized >2014-08-03 01:24:23,759 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-24) [41bfb6e1] Attempting to update VMs/Templates Ovf. >2014-08-03 01:31:40,435 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-19) domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 in problem. vds: rack >2014-08-03 01:31:40,437 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-19) domain 44559ec0-6bef-491c-bbd6-e67278f37987:export in problem. vds: rack >2014-08-03 01:31:55,560 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-25) Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 recovered from problem. vds: rack >2014-08-03 01:31:55,561 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-25) Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 has recovered from problem. No active host in the DC is reporting it as problematic, so clearing the domain recovery timer. >2014-08-03 01:31:55,564 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-25) Domain 44559ec0-6bef-491c-bbd6-e67278f37987:export recovered from problem. vds: rack >2014-08-03 01:31:55,565 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-25) Domain 44559ec0-6bef-491c-bbd6-e67278f37987:export has recovered from problem. No active host in the DC is reporting it as problematic, so clearing the domain recovery timer. >2014-08-03 01:36:27,968 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-28) domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 in problem. vds: rack >2014-08-03 01:36:27,969 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-28) domain 44559ec0-6bef-491c-bbd6-e67278f37987:export in problem. vds: rack >2014-08-03 01:39:17,588 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-53) Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2014-08-03 01:39:17,592 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-53) [3c2dfb77] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-08-03 01:39:17,596 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-53) [3c2dfb77] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Active to Unknown, reason : null >2014-08-03 01:39:17,598 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-53) [3c2dfb77] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Active to Unknown, reason : null >2014-08-03 01:39:17,612 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-53) [3c2dfb77] Correlation ID: 3c2dfb77, Call Stack: null, Custom Event ID: -1, Message: Invalid status on Data Center Default. Setting Data Center status to Non Responsive (On host rack, Error: Cannot access master storage domain). >2014-08-03 01:41:17,640 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-53) [3c2dfb77] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2014-08-03 01:41:17,645 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-53) [3c2dfb77] hostFromVds::selectedVds - rack, spmStatus returned null! >2014-08-03 01:41:17,646 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-53) [3c2dfb77] IrsBroker::Failed::GetStoragePoolInfoVDS >2014-08-03 01:41:17,647 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-53) [3c2dfb77] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2014-08-03 01:41:17,682 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-8) [3c268f29] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage >2014-08-03 01:41:17,700 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-8) [3c268f29] START, ResetIrsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, ignoreStopFailed = true), log id: 51a0fdcc >2014-08-03 01:41:17,703 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-8) [3c268f29] START, SpmStopVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9), log id: 3efb9be1 >2014-08-03 01:41:17,709 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-8) [3c268f29] SpmStopVDSCommand::Stopping SPM on vds rack, pool id 00000002-0002-0002-0002-0000000001d9 >2014-08-03 01:43:17,611 WARN [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-8) [3c268f29] could not stop spm of pool 00000002-0002-0002-0002-0000000001d9 on vds 17f7fe56-5f32-40ef-b098-611b84484e0b - reason: org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcRunTimeException: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException >2014-08-03 01:43:17,613 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-8) [3c268f29] FINISH, SpmStopVDSCommand, log id: 3efb9be1 >2014-08-03 01:43:17,618 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-24) [41bfb6e1] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: VDSNetworkException: org.apache.commons.httpclient.NoHttpResponseException: The server 10.34.62.208 failed to respond >2014-08-03 01:43:17,620 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-24) [41bfb6e1] hostFromVds::selectedVds - rack, spmStatus returned null! >2014-08-03 01:43:17,623 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetStatsVDSCommand] (DefaultQuartzScheduler_Worker-64) [686e64a1] Command GetStatsVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-08-03 01:43:17,626 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-64) [686e64a1] Host rack is not responding. It will stay in Connecting state for a grace period of $80 seconds and after that an attempt to fence the host will be issued. >2014-08-03 01:43:17,628 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-64) [686e64a1] vds::refreshVdsStats Failed getVdsStats, vds = 17f7fe56-5f32-40ef-b098-611b84484e0b : rack, error = VDSNetworkException: java.net.ConnectException: Connection refused >2014-08-03 01:43:17,630 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-8) [3c268f29] FINISH, ResetIrsVDSCommand, log id: 51a0fdcc >2014-08-03 01:43:17,629 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-64) [686e64a1] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsStats(VdsUpdateRunTimeInfo.java:669) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:623) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:235) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) [:1.7.0_55] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_55] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_55] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_55] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_55] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_55] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_55] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_55] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy82.getVdsStats(Unknown Source) > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_55] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_55] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_55] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_55] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_55] > >2014-08-03 01:43:17,640 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-8) [3c268f29] Correlation ID: 3c268f29, Job ID: 12ffb774-0b76-4f7f-a006-17a9047fcc90, Call Stack: null, Custom Event ID: -1, Message: Failed to Reconstruct Master Domain for Data Center Default. >2014-08-03 01:43:17,658 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-64) [686e64a1] Failed to refresh VDS , vds = 17f7fe56-5f32-40ef-b098-611b84484e0b : rack, VDS Network Error, continuing. >java.net.ConnectException: Connection refused >2014-08-03 01:43:17,659 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-34) [3c2dfb77] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue >2014-08-03 01:43:17,661 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-9) starting processDomainRecovery for domain 44559ec0-6bef-491c-bbd6-e67278f37987:export >2014-08-03 01:43:17,730 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-9) Domain 44559ec0-6bef-491c-bbd6-e67278f37987:export was reported by all hosts in status UP as problematic. Moving the domain to NonOperational. >2014-08-03 01:43:17,746 INFO [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-9) [78f1ddd1] Lock Acquired to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE >, sharedLocks= ] >2014-08-03 01:43:17,753 INFO [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-9) [78f1ddd1] Running command: DeactivateStorageDomainCommand internal: true. Entities affected : ID: 44559ec0-6bef-491c-bbd6-e67278f37987 Type: Storage >2014-08-03 01:43:17,759 INFO [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-9) [78f1ddd1] Lock freed to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE >, sharedLocks= ] >2014-08-03 01:43:17,770 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-9) [78f1ddd1] Correlation ID: 78f1ddd1, Job ID: 26bd13d4-17ac-45b5-a9c8-a297362db7f8, Call Stack: null, Custom Event ID: -1, Message: Storage Domain export (Data Center Default) was deactivated by system because it's not visible by any of the hosts. >2014-08-03 01:43:17,775 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-9) [78f1ddd1] Removing vds [] from the domain in maintenance cache >2014-08-03 01:43:17,776 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-9) [78f1ddd1] Removing host(s) [] from hosts unseen domain report cache >2014-08-03 01:43:17,777 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-50) starting processDomainRecovery for domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 >2014-08-03 01:43:17,782 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-50) Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 was reported by all hosts in status UP as problematic. Not moving the domain to NonOperational because it is being reconstructed now. >2014-08-03 01:43:17,800 WARN [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-50) [7f4e3a8b] CanDoAction of action ReconstructMasterDomain failed. Reasons:VAR__ACTION__RECONSTRUCT_MASTER,VAR__TYPE__STORAGE__DOMAIN,ACTION_TYPE_FAILED_NO_VDS_IN_POOL >2014-08-03 01:43:17,807 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-50) [7f4e3a8b] Removing vds [] from the domain in maintenance cache >2014-08-03 01:43:17,808 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-50) [7f4e3a8b] Removing host(s) [] from hosts unseen domain report cache >2014-08-03 01:43:17,809 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-34) [3c2dfb77] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue >2014-08-03 01:43:21,108 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-9) START, GetHardwareInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]), log id: 57cdbcee >2014-08-03 01:43:21,163 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-9) FINISH, GetHardwareInfoVDSCommand, log id: 57cdbcee >2014-08-03 01:43:21,188 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-9) [1f9b3de5] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS >2014-08-03 01:43:21,238 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-9) [1d78fb26] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-08-03 01:43:21,246 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-9) [311da2ef] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-08-03 01:43:21,258 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-9) [311da2ef] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, storageType = NFS, connectionList = [{ id: 05047c9a-152e-4ef2-b118-4a1753c1eae0, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35export, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 6201d59d >2014-08-03 01:43:21,320 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-9) [311da2ef] FINISH, ConnectStorageServerVDSCommand, return: {5083f370-af09-457b-9b73-3440e8b65a07=0, 05047c9a-152e-4ef2-b118-4a1753c1eae0=0}, log id: 6201d59d >2014-08-03 01:43:21,321 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-9) [311da2ef] Host rack storage connection was succeeded >2014-08-03 01:43:21,328 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-12) START, ConnectStoragePoolVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, masterVersion = 1), log id: 1a26ec0c >2014-08-03 01:43:27,631 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-17) Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Inactive to Unknown, reason : no reporting hosts >2014-08-03 01:43:27,634 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-17) Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Inactive to Unknown, reason : no reporting hosts >2014-08-03 01:44:21,726 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-12) FINISH, ConnectStoragePoolVDSCommand, log id: 1a26ec0c >2014-08-03 01:44:21,774 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-9) [311da2ef] START, SetMOMPolicyParametersVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 42a37983 >2014-08-03 01:44:21,869 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-9) [311da2ef] FINISH, SetMOMPolicyParametersVDSCommand, log id: 42a37983 >2014-08-03 01:44:21,880 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-9) [311da2ef] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host rack was set to Up. >2014-08-03 01:44:21,887 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-9) [311da2ef] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host rack. >2014-08-03 01:44:21,906 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-9) [aa1055d] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS >2014-08-03 01:44:27,774 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-79) [37510499] hostFromVds::selectedVds - rack, spmStatus Free, storage pool Default >2014-08-03 01:44:27,780 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-79) [37510499] starting spm on vds rack, storage pool Default, prevId -1, LVER -1 >2014-08-03 01:44:27,782 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-79) [37510499] START, SpmStartVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, prevId=-1, prevLVER=-1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 70b313e7 >2014-08-03 01:44:27,830 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-79) [37510499] spmStart polling started: taskId = af93a1cf-cc2f-4df7-8929-868095e457a5 >2014-08-03 01:44:28,837 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-79) [37510499] spmStart polling ended: taskId = af93a1cf-cc2f-4df7-8929-868095e457a5 task status = finished >2014-08-03 01:44:28,893 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-79) [37510499] spmStart polling ended, spm status: SPM >2014-08-03 01:44:28,895 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-79) [37510499] START, HSMClearTaskVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, taskId=af93a1cf-cc2f-4df7-8929-868095e457a5), log id: 1ef7fc4e >2014-08-03 01:44:28,940 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-79) [37510499] FINISH, HSMClearTaskVDSCommand, log id: 1ef7fc4e >2014-08-03 01:44:28,941 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-79) [37510499] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@51dd38bb, log id: 70b313e7 >2014-08-03 01:44:28,945 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-79) [37510499] Initialize Irs proxy from vds: 10.34.62.208 >2014-08-03 01:44:28,951 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-79) [37510499] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208). >2014-08-03 01:44:28,960 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-26) [37510499] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 3dd473ae >2014-08-03 01:44:28,995 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-26) [37510499] -- executeIrsBrokerCommand: Attempting on storage pool 00000002-0002-0002-0002-0000000001d9 >2014-08-03 01:44:28,997 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-26) [37510499] START, HSMGetAllTasksInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 42baa90c >2014-08-03 01:44:29,002 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-26) [37510499] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 42baa90c >2014-08-03 01:44:29,003 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-26) [37510499] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 3dd473ae >2014-08-03 01:44:29,004 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (org.ovirt.thread.pool-8-thread-26) [37510499] Discovered no tasks on Storage Pool Default >2014-08-03 02:23:55,750 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-25) [50daf5eb] Principal admin::internal synchronized >2014-08-03 02:24:23,791 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-30) [dd7f8b1] Attempting to update VMs/Templates Ovf. >2014-08-03 03:23:55,760 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-91) [64566e87] Principal admin::internal synchronized >2014-08-03 03:24:23,827 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-83) Attempting to update VMs/Templates Ovf. >2014-08-03 03:35:35,000 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-47) Start deleteAgedOutAuditLogs >2014-08-03 03:35:35,001 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-67) [6199900c] Start deleteAgedOutCommandEntities >2014-08-03 03:35:35,004 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-47) Finished deleteAgedOutAuditLogs >2014-08-03 03:35:35,007 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-67) [6199900c] Finished deleteAgedOutCommandEntities >2014-08-03 04:23:55,772 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-24) [41bfb6e1] Principal admin::internal synchronized >2014-08-03 04:24:23,860 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-98) [5aa74089] Attempting to update VMs/Templates Ovf. >2014-08-03 04:39:14,569 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-19) domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 in problem. vds: rack >2014-08-03 04:39:14,571 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-19) domain 44559ec0-6bef-491c-bbd6-e67278f37987:export in problem. vds: rack >2014-08-03 04:39:44,844 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-12) Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 recovered from problem. vds: rack >2014-08-03 04:39:44,845 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-12) Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 has recovered from problem. No active host in the DC is reporting it as problematic, so clearing the domain recovery timer. >2014-08-03 04:39:44,848 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-12) Domain 44559ec0-6bef-491c-bbd6-e67278f37987:export recovered from problem. vds: rack >2014-08-03 04:39:44,849 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-12) Domain 44559ec0-6bef-491c-bbd6-e67278f37987:export has recovered from problem. No active host in the DC is reporting it as problematic, so clearing the domain recovery timer. >2014-08-03 04:44:17,058 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-14) domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 in problem. vds: rack >2014-08-03 04:44:17,060 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-14) domain 44559ec0-6bef-491c-bbd6-e67278f37987:export in problem. vds: rack >2014-08-03 04:46:52,473 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-52) [2ccfb0f3] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2014-08-03 04:46:52,478 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-52) [8557062] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-08-03 04:46:52,482 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-52) [8557062] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Active to Unknown, reason : null >2014-08-03 04:46:52,484 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-52) [8557062] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Active to Unknown, reason : null >2014-08-03 04:46:52,495 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-52) [8557062] Correlation ID: 8557062, Call Stack: null, Custom Event ID: -1, Message: Invalid status on Data Center Default. Setting Data Center status to Non Responsive (On host rack, Error: Cannot access master storage domain). >2014-08-03 04:48:52,526 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-52) [8557062] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2014-08-03 04:48:52,530 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-52) [8557062] hostFromVds::selectedVds - rack, spmStatus returned null! >2014-08-03 04:48:52,531 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-52) [8557062] IrsBroker::Failed::GetStoragePoolInfoVDS >2014-08-03 04:48:52,532 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-52) [8557062] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2014-08-03 04:48:52,579 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-20) [766b439f] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage >2014-08-03 04:48:52,598 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-20) [766b439f] START, ResetIrsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, ignoreStopFailed = true), log id: 5d22bd8 >2014-08-03 04:48:52,602 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-20) [766b439f] START, SpmStopVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9), log id: 40ecc90d >2014-08-03 04:48:52,608 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-20) [766b439f] SpmStopVDSCommand::Stopping SPM on vds rack, pool id 00000002-0002-0002-0002-0000000001d9 >2014-08-03 04:50:00,010 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-24) [41bfb6e1] Autorecovering 1 storage domains >2014-08-03 04:50:00,011 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-24) [41bfb6e1] Autorecovering storage domains id: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 >2014-08-03 04:50:00,013 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-24) [4380dba3] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage >2014-08-03 04:50:00,014 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-24) [4380dba3] ConnectDomainToStorage. Before Connect all hosts to pool. Time:8/3/14 4:50 AM >2014-08-03 04:50:00,025 INFO [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-8-thread-8) [7a385e6e] Running command: ConnectStorageToVdsCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2014-08-03 04:50:00,029 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-8) [7a385e6e] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 684c00d3 >2014-08-03 04:50:52,502 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-11) [aeb0141] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2014-08-03 04:50:52,511 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-11) [aeb0141] hostFromVds::selectedVds - rack, spmStatus returned null! >2014-08-03 04:50:52,513 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-11) [aeb0141] IrsBroker::Failed::GetStoragePoolInfoVDS >2014-08-03 04:50:52,516 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-11) [aeb0141] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2014-08-03 04:51:52,504 WARN [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-20) [766b439f] could not stop spm of pool 00000002-0002-0002-0002-0000000001d9 on vds 17f7fe56-5f32-40ef-b098-611b84484e0b - reason: org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcRunTimeException: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException >2014-08-03 04:51:52,506 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-20) [766b439f] FINISH, SpmStopVDSCommand, log id: 40ecc90d >2014-08-03 04:51:52,508 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-8) [7a385e6e] Command ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]) execution failed. Exception: VDSNetworkException: org.apache.commons.httpclient.NoHttpResponseException: The server 10.34.62.208 failed to respond >2014-08-03 04:51:52,513 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-88) Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-08-03 04:51:52,512 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetAllVmStatsVDSCommand] (DefaultQuartzScheduler_Worker-5) Command GetAllVmStatsVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-08-03 04:51:52,519 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-88) hostFromVds::selectedVds - rack, spmStatus returned null! >2014-08-03 04:51:52,519 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-5) Host rack is not responding. It will stay in Connecting state for a grace period of $80 seconds and after that an attempt to fence the host will be issued. >2014-08-03 04:51:52,519 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-8) [7a385e6e] FINISH, ConnectStorageServerVDSCommand, log id: 684c00d3 >2014-08-03 04:51:52,521 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-20) [766b439f] FINISH, ResetIrsVDSCommand, log id: 5d22bd8 >2014-08-03 04:51:52,522 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-5) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:406) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.fetchRunningVms(VdsUpdateRunTimeInfo.java:971) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVmStats(VdsUpdateRunTimeInfo.java:920) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:638) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:235) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) [:1.7.0_55] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_55] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_55] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_55] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_55] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_55] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_55] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_55] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy82.getAllVmStats(Unknown Source) > at sun.reflect.GeneratedMethodAccessor25.invoke(Unknown Source) [:1.7.0_55] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_55] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_55] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_55] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_55] > >2014-08-03 04:51:52,534 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-20) [766b439f] Correlation ID: 766b439f, Job ID: 0daff75a-6375-4171-89ce-94b17606f321, Call Stack: null, Custom Event ID: -1, Message: Failed to Reconstruct Master Domain for Data Center Default. >2014-08-03 04:51:52,525 ERROR [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-8-thread-8) [7a385e6e] Command org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: org.apache.commons.httpclient.NoHttpResponseException: The server 10.34.62.208 failed to respond (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-08-03 04:51:52,559 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-14) [8557062] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue >2014-08-03 04:51:52,576 ERROR [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-8-thread-8) [7a385e6e] Transaction rolled-back for command: org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand. >2014-08-03 04:51:52,579 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-24) [4380dba3] ConnectDomainToStorage. After Connect all hosts to pool. Time:8/3/14 4:51 AM >2014-08-03 04:51:52,588 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-43) starting processDomainRecovery for domain 44559ec0-6bef-491c-bbd6-e67278f37987:export >2014-08-03 04:51:52,600 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-5) Failed to refresh VDS , vds = 17f7fe56-5f32-40ef-b098-611b84484e0b : rack, VDS Network Error, continuing. >java.net.ConnectException: Connection refused >2014-08-03 04:51:52,672 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-43) Domain 44559ec0-6bef-491c-bbd6-e67278f37987:export was reported by all hosts in status UP as problematic. Moving the domain to NonOperational. >2014-08-03 04:51:52,727 INFO [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-43) [472f3970] Lock Acquired to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE >, sharedLocks= ] >2014-08-03 04:51:52,735 INFO [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-43) [472f3970] Running command: DeactivateStorageDomainCommand internal: true. Entities affected : ID: 44559ec0-6bef-491c-bbd6-e67278f37987 Type: Storage >2014-08-03 04:51:52,742 INFO [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-43) [472f3970] Lock freed to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE >, sharedLocks= ] >2014-08-03 04:51:52,755 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-43) [472f3970] Correlation ID: 472f3970, Job ID: 9a3762fb-248b-4198-b8c0-6fcf3ea4f970, Call Stack: null, Custom Event ID: -1, Message: Storage Domain export (Data Center Default) was deactivated by system because it's not visible by any of the hosts. >2014-08-03 04:51:52,762 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-43) [472f3970] Removing vds [] from the domain in maintenance cache >2014-08-03 04:51:52,763 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-43) [472f3970] Removing host(s) [] from hosts unseen domain report cache >2014-08-03 04:51:52,776 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-50) starting processDomainRecovery for domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 >2014-08-03 04:51:52,782 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-50) Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 was reported by all hosts in status UP as problematic. Not moving the domain to NonOperational because it is being reconstructed now. >2014-08-03 04:51:52,812 WARN [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-50) [1c85032d] CanDoAction of action ReconstructMasterDomain failed. Reasons:VAR__ACTION__RECONSTRUCT_MASTER,VAR__TYPE__STORAGE__DOMAIN,ACTION_TYPE_FAILED_NO_VDS_IN_POOL >2014-08-03 04:51:52,821 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-50) [1c85032d] Removing vds [] from the domain in maintenance cache >2014-08-03 04:51:52,823 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-50) [1c85032d] Removing host(s) [] from hosts unseen domain report cache >2014-08-03 04:51:52,825 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-14) [8557062] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue >2014-08-03 04:51:55,960 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-93) [156fe0ca] START, GetHardwareInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]), log id: 18e5c347 >2014-08-03 04:51:56,011 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-93) [156fe0ca] FINISH, GetHardwareInfoVDSCommand, log id: 18e5c347 >2014-08-03 04:51:56,053 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-93) [5b8094aa] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS >2014-08-03 04:51:56,068 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-93) [11555cf2] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-08-03 04:51:56,078 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-93) [18923d9] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-08-03 04:51:56,090 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-93) [18923d9] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, storageType = NFS, connectionList = [{ id: 05047c9a-152e-4ef2-b118-4a1753c1eae0, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35export, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 314f5dea >2014-08-03 04:51:56,150 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-93) [18923d9] FINISH, ConnectStorageServerVDSCommand, return: {5083f370-af09-457b-9b73-3440e8b65a07=0, 05047c9a-152e-4ef2-b118-4a1753c1eae0=0}, log id: 314f5dea >2014-08-03 04:51:56,151 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-93) [18923d9] Host rack storage connection was succeeded >2014-08-03 04:51:56,160 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-10) START, ConnectStoragePoolVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, masterVersion = 1), log id: 46b0eb76 >2014-08-03 04:52:02,524 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-79) [37510499] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Inactive to Unknown, reason : no reporting hosts >2014-08-03 04:52:02,528 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-79) [37510499] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Inactive to Unknown, reason : no reporting hosts >2014-08-03 04:52:56,566 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-10) FINISH, ConnectStoragePoolVDSCommand, log id: 46b0eb76 >2014-08-03 04:52:56,608 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-93) [18923d9] START, SetMOMPolicyParametersVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 66590958 >2014-08-03 04:52:56,701 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-93) [18923d9] FINISH, SetMOMPolicyParametersVDSCommand, log id: 66590958 >2014-08-03 04:52:56,709 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-93) [18923d9] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host rack was set to Up. >2014-08-03 04:52:56,717 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-93) [18923d9] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host rack. >2014-08-03 04:52:56,740 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-93) [71be4699] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS >2014-08-03 04:53:02,727 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-40) hostFromVds::selectedVds - rack, spmStatus Free, storage pool Default >2014-08-03 04:53:02,733 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-40) starting spm on vds rack, storage pool Default, prevId -1, LVER -1 >2014-08-03 04:53:02,735 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-40) START, SpmStartVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, prevId=-1, prevLVER=-1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 40d4557f >2014-08-03 04:53:02,784 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-40) spmStart polling started: taskId = 30cf0799-5a02-40eb-99c2-41c284129530 >2014-08-03 04:53:03,794 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-40) spmStart polling ended: taskId = 30cf0799-5a02-40eb-99c2-41c284129530 task status = finished >2014-08-03 04:53:03,849 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-40) spmStart polling ended, spm status: SPM >2014-08-03 04:53:03,851 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-40) START, HSMClearTaskVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, taskId=30cf0799-5a02-40eb-99c2-41c284129530), log id: 1e496c0f >2014-08-03 04:53:03,896 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-40) FINISH, HSMClearTaskVDSCommand, log id: 1e496c0f >2014-08-03 04:53:03,897 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-40) FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@439b8a20, log id: 40d4557f >2014-08-03 04:53:03,901 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-40) Initialize Irs proxy from vds: 10.34.62.208 >2014-08-03 04:53:03,907 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-40) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208). >2014-08-03 04:53:03,917 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-32) START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 18dc5e46 >2014-08-03 04:53:03,953 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-32) -- executeIrsBrokerCommand: Attempting on storage pool 00000002-0002-0002-0002-0000000001d9 >2014-08-03 04:53:03,955 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-32) START, HSMGetAllTasksInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 7066ebad >2014-08-03 04:53:03,960 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-32) FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 7066ebad >2014-08-03 04:53:03,961 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-32) FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 18dc5e46 >2014-08-03 04:53:03,962 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (org.ovirt.thread.pool-8-thread-32) Discovered no tasks on Storage Pool Default >2014-08-03 05:23:55,778 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-72) Principal admin::internal synchronized >2014-08-03 05:24:23,877 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-42) [161c5527] Attempting to update VMs/Templates Ovf. >2014-08-03 06:23:55,785 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-35) [5b4b3aec] Principal admin::internal synchronized >2014-08-03 06:24:23,360 WARN [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-8-thread-36) Executing a command: java.util.concurrent.FutureTask , but note that there are 1 tasks in the queue. >2014-08-03 06:24:23,894 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-26) [6bd9692] Attempting to update VMs/Templates Ovf. >2014-08-03 07:23:55,791 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-93) [71be4699] Principal admin::internal synchronized >2014-08-03 07:24:23,908 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-38) [1b9ef0f] Attempting to update VMs/Templates Ovf. >2014-08-03 07:52:51,484 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-29) domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 in problem. vds: rack >2014-08-03 07:52:51,486 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-29) domain 44559ec0-6bef-491c-bbd6-e67278f37987:export in problem. vds: rack >2014-08-03 07:54:18,522 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-24) [4380dba3] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2014-08-03 07:54:18,527 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-24) [61a1f9b5] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-08-03 07:54:18,531 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-24) [61a1f9b5] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Active to Unknown, reason : null >2014-08-03 07:54:18,533 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-24) [61a1f9b5] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Active to Unknown, reason : null >2014-08-03 07:54:18,549 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-24) [61a1f9b5] Correlation ID: 61a1f9b5, Call Stack: null, Custom Event ID: -1, Message: Invalid status on Data Center Default. Setting Data Center status to Non Responsive (On host rack, Error: Cannot access master storage domain). >2014-08-03 07:56:18,576 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-24) [61a1f9b5] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2014-08-03 07:56:18,579 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-24) [61a1f9b5] hostFromVds::selectedVds - rack, spmStatus returned null! >2014-08-03 07:56:18,580 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-24) [61a1f9b5] IrsBroker::Failed::GetStoragePoolInfoVDS >2014-08-03 07:56:18,581 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-24) [61a1f9b5] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2014-08-03 07:56:18,621 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-27) [242d9d56] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage >2014-08-03 07:56:18,640 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-27) [242d9d56] START, ResetIrsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, ignoreStopFailed = true), log id: d41f226 >2014-08-03 07:56:18,646 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-27) [242d9d56] START, SpmStopVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9), log id: 1b1e2478 >2014-08-03 07:56:18,652 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-27) [242d9d56] SpmStopVDSCommand::Stopping SPM on vds rack, pool id 00000002-0002-0002-0002-0000000001d9 >2014-08-03 07:58:18,546 WARN [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-27) [242d9d56] could not stop spm of pool 00000002-0002-0002-0002-0000000001d9 on vds 17f7fe56-5f32-40ef-b098-611b84484e0b - reason: org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcRunTimeException: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException >2014-08-03 07:58:18,550 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-27) [242d9d56] FINISH, SpmStopVDSCommand, log id: 1b1e2478 >2014-08-03 07:58:18,553 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ListVDSCommand] (DefaultQuartzScheduler_Worker-6) [7f1e369e] Command ListVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-08-03 07:58:18,555 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-40) Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: VDSNetworkException: org.apache.commons.httpclient.NoHttpResponseException: The server 10.34.62.208 failed to respond >2014-08-03 07:58:18,560 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-6) [7f1e369e] Host rack is not responding. It will stay in Connecting state for a grace period of $80 seconds and after that an attempt to fence the host will be issued. >2014-08-03 07:58:18,563 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-27) [242d9d56] FINISH, ResetIrsVDSCommand, log id: d41f226 >2014-08-03 07:58:18,561 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-40) hostFromVds::selectedVds - rack, spmStatus returned null! >2014-08-03 07:58:18,563 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-6) [7f1e369e] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:406) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.fetchRunningVms(VdsUpdateRunTimeInfo.java:971) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVmStats(VdsUpdateRunTimeInfo.java:920) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:638) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:235) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) [:1.7.0_55] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_55] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_55] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_55] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_55] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_55] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_55] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_55] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy82.list(Unknown Source) > at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source) [:1.7.0_55] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_55] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_55] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_55] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_55] > >2014-08-03 07:58:18,577 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-27) [242d9d56] Correlation ID: 242d9d56, Job ID: c1a40082-c940-4518-b450-ec68c25d1d62, Call Stack: null, Custom Event ID: -1, Message: Failed to Reconstruct Master Domain for Data Center Default. >2014-08-03 07:58:18,603 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-6) [7f1e369e] Failed to refresh VDS , vds = 17f7fe56-5f32-40ef-b098-611b84484e0b : rack, VDS Network Error, continuing. >java.net.ConnectException: Connection refused >2014-08-03 07:58:18,607 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-18) [61a1f9b5] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue >2014-08-03 07:58:18,609 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-36) starting processDomainRecovery for domain 44559ec0-6bef-491c-bbd6-e67278f37987:export >2014-08-03 07:58:18,680 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-36) Domain 44559ec0-6bef-491c-bbd6-e67278f37987:export was reported by all hosts in status UP as problematic. Moving the domain to NonOperational. >2014-08-03 07:58:18,703 INFO [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-36) [536db155] Lock Acquired to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE >, sharedLocks= ] >2014-08-03 07:58:18,710 INFO [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-36) [536db155] Running command: DeactivateStorageDomainCommand internal: true. Entities affected : ID: 44559ec0-6bef-491c-bbd6-e67278f37987 Type: Storage >2014-08-03 07:58:18,717 INFO [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-36) [536db155] Lock freed to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE >, sharedLocks= ] >2014-08-03 07:58:18,728 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-36) [536db155] Correlation ID: 536db155, Job ID: a595a0b5-bf76-45ee-b43c-436637c60038, Call Stack: null, Custom Event ID: -1, Message: Storage Domain export (Data Center Default) was deactivated by system because it's not visible by any of the hosts. >2014-08-03 07:58:18,734 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-36) [536db155] Removing vds [] from the domain in maintenance cache >2014-08-03 07:58:18,735 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-36) [536db155] Removing host(s) [] from hosts unseen domain report cache >2014-08-03 07:58:18,737 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-41) starting processDomainRecovery for domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 >2014-08-03 07:58:18,741 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-41) Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 was reported by all hosts in status UP as problematic. Not moving the domain to NonOperational because it is being reconstructed now. >2014-08-03 07:58:18,766 WARN [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-41) [5bb83e6d] CanDoAction of action ReconstructMasterDomain failed. Reasons:VAR__ACTION__RECONSTRUCT_MASTER,VAR__TYPE__STORAGE__DOMAIN,ACTION_TYPE_FAILED_NO_VDS_IN_POOL >2014-08-03 07:58:18,774 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-41) [5bb83e6d] Removing vds [] from the domain in maintenance cache >2014-08-03 07:58:18,775 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-41) [5bb83e6d] Removing host(s) [] from hosts unseen domain report cache >2014-08-03 07:58:18,776 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-18) [61a1f9b5] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue >2014-08-03 07:58:21,872 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-1) START, GetHardwareInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]), log id: 4d09556a >2014-08-03 07:58:21,924 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-1) FINISH, GetHardwareInfoVDSCommand, log id: 4d09556a >2014-08-03 07:58:21,963 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-1) [6b7975ba] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS >2014-08-03 07:58:21,973 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-1) [6dd1a1f8] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-08-03 07:58:21,980 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-1) [2e6177fb] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-08-03 07:58:21,992 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-1) [2e6177fb] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, storageType = NFS, connectionList = [{ id: 05047c9a-152e-4ef2-b118-4a1753c1eae0, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35export, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 755a37c7 >2014-08-03 07:58:22,055 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-1) [2e6177fb] FINISH, ConnectStorageServerVDSCommand, return: {5083f370-af09-457b-9b73-3440e8b65a07=0, 05047c9a-152e-4ef2-b118-4a1753c1eae0=0}, log id: 755a37c7 >2014-08-03 07:58:22,057 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-1) [2e6177fb] Host rack storage connection was succeeded >2014-08-03 07:58:22,067 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-20) START, ConnectStoragePoolVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, masterVersion = 1), log id: 6b85978e >2014-08-03 07:58:28,570 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-78) Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Inactive to Unknown, reason : no reporting hosts >2014-08-03 07:58:28,574 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-78) Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Inactive to Unknown, reason : no reporting hosts >2014-08-03 07:59:22,463 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-20) FINISH, ConnectStoragePoolVDSCommand, log id: 6b85978e >2014-08-03 07:59:22,476 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-1) [2e6177fb] START, SetMOMPolicyParametersVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 73c51cbb >2014-08-03 07:59:22,570 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-1) [2e6177fb] FINISH, SetMOMPolicyParametersVDSCommand, log id: 73c51cbb >2014-08-03 07:59:22,578 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-1) [2e6177fb] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host rack was set to Up. >2014-08-03 07:59:22,584 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-1) [2e6177fb] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host rack. >2014-08-03 07:59:22,599 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-1) [795e8140] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS >2014-08-03 07:59:28,725 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-62) [2cdbf22e] hostFromVds::selectedVds - rack, spmStatus Free, storage pool Default >2014-08-03 07:59:28,732 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-62) [2cdbf22e] starting spm on vds rack, storage pool Default, prevId -1, LVER -1 >2014-08-03 07:59:28,734 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-62) [2cdbf22e] START, SpmStartVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, prevId=-1, prevLVER=-1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 47334e56 >2014-08-03 07:59:28,783 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-62) [2cdbf22e] spmStart polling started: taskId = 3c17a062-6669-495e-8dbb-5b48f9b758d4 >2014-08-03 07:59:29,789 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-62) [2cdbf22e] spmStart polling ended: taskId = 3c17a062-6669-495e-8dbb-5b48f9b758d4 task status = finished >2014-08-03 07:59:29,846 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-62) [2cdbf22e] spmStart polling ended, spm status: SPM >2014-08-03 07:59:29,848 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-62) [2cdbf22e] START, HSMClearTaskVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, taskId=3c17a062-6669-495e-8dbb-5b48f9b758d4), log id: 1e65786d >2014-08-03 07:59:29,893 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-62) [2cdbf22e] FINISH, HSMClearTaskVDSCommand, log id: 1e65786d >2014-08-03 07:59:29,894 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-62) [2cdbf22e] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@234cd6d3, log id: 47334e56 >2014-08-03 07:59:29,898 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-62) [2cdbf22e] Initialize Irs proxy from vds: 10.34.62.208 >2014-08-03 07:59:29,905 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-62) [2cdbf22e] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208). >2014-08-03 07:59:29,915 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-1) [2cdbf22e] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 5f7423dd >2014-08-03 07:59:29,952 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-1) [2cdbf22e] -- executeIrsBrokerCommand: Attempting on storage pool 00000002-0002-0002-0002-0000000001d9 >2014-08-03 07:59:29,955 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-1) [2cdbf22e] START, HSMGetAllTasksInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 7e4e0a7c >2014-08-03 07:59:29,962 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-1) [2cdbf22e] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 7e4e0a7c >2014-08-03 07:59:29,963 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-1) [2cdbf22e] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 5f7423dd >2014-08-03 07:59:29,964 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (org.ovirt.thread.pool-8-thread-1) [2cdbf22e] Discovered no tasks on Storage Pool Default >2014-08-03 08:23:55,797 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-80) [78845858] Principal admin::internal synchronized >2014-08-03 08:24:23,921 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-21) [81f884c] Attempting to update VMs/Templates Ovf. >2014-08-03 09:23:55,806 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-26) [6bd9692] Principal admin::internal synchronized >2014-08-03 09:24:23,954 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-4) Attempting to update VMs/Templates Ovf. >2014-08-03 10:23:55,812 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-10) Principal admin::internal synchronized >2014-08-03 10:24:23,969 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-99) Attempting to update VMs/Templates Ovf. >2014-08-03 10:59:11,542 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-38) domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 in problem. vds: rack >2014-08-03 10:59:11,544 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-38) domain 44559ec0-6bef-491c-bbd6-e67278f37987:export in problem. vds: rack >2014-08-03 11:00:45,458 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-70) [593e74a7] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2014-08-03 11:00:45,462 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-70) [5f00c7bf] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-08-03 11:00:45,466 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-70) [5f00c7bf] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Active to Unknown, reason : null >2014-08-03 11:00:45,468 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-70) [5f00c7bf] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Active to Unknown, reason : null >2014-08-03 11:00:45,479 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-70) [5f00c7bf] Correlation ID: 5f00c7bf, Call Stack: null, Custom Event ID: -1, Message: Invalid status on Data Center Default. Setting Data Center status to Non Responsive (On host rack, Error: Cannot access master storage domain). >2014-08-03 11:02:45,508 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-70) [5f00c7bf] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2014-08-03 11:02:45,513 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-70) [5f00c7bf] hostFromVds::selectedVds - rack, spmStatus returned null! >2014-08-03 11:02:45,515 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-70) [5f00c7bf] IrsBroker::Failed::GetStoragePoolInfoVDS >2014-08-03 11:02:45,516 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-70) [5f00c7bf] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2014-08-03 11:02:45,551 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-48) [287a2335] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage >2014-08-03 11:02:45,567 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-48) [287a2335] START, ResetIrsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, ignoreStopFailed = true), log id: 612569f0 >2014-08-03 11:02:45,571 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-48) [287a2335] START, SpmStopVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9), log id: 2458fd84 >2014-08-03 11:02:45,576 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-48) [287a2335] SpmStopVDSCommand::Stopping SPM on vds rack, pool id 00000002-0002-0002-0002-0000000001d9 >2014-08-03 11:04:45,480 WARN [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-48) [287a2335] could not stop spm of pool 00000002-0002-0002-0002-0000000001d9 on vds 17f7fe56-5f32-40ef-b098-611b84484e0b - reason: org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcRunTimeException: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException >2014-08-03 11:04:45,482 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-48) [287a2335] FINISH, SpmStopVDSCommand, log id: 2458fd84 >2014-08-03 11:04:45,487 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetStatsVDSCommand] (DefaultQuartzScheduler_Worker-23) [1802a60f] Command GetStatsVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-08-03 11:04:45,490 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-35) [5b4b3aec] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: VDSNetworkException: org.apache.commons.httpclient.NoHttpResponseException: The server 10.34.62.208 failed to respond >2014-08-03 11:04:45,497 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-23) [1802a60f] Host rack is not responding. It will stay in Connecting state for a grace period of $80 seconds and after that an attempt to fence the host will be issued. >2014-08-03 11:04:45,499 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-48) [287a2335] FINISH, ResetIrsVDSCommand, log id: 612569f0 >2014-08-03 11:04:45,497 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-35) [5b4b3aec] hostFromVds::selectedVds - rack, spmStatus returned null! >2014-08-03 11:04:45,500 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-23) [1802a60f] vds::refreshVdsStats Failed getVdsStats, vds = 17f7fe56-5f32-40ef-b098-611b84484e0b : rack, error = VDSNetworkException: java.net.ConnectException: Connection refused >2014-08-03 11:04:45,504 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-23) [1802a60f] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsStats(VdsUpdateRunTimeInfo.java:669) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:623) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:235) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) [:1.7.0_55] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_55] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_55] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_55] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_55] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_55] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_55] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_55] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy82.getVdsStats(Unknown Source) > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_55] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_55] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_55] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_55] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_55] > >2014-08-03 11:04:45,518 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-48) [287a2335] Correlation ID: 287a2335, Job ID: e32f59b0-6a5e-4c1f-9e0d-32ec64fb4f45, Call Stack: null, Custom Event ID: -1, Message: Failed to Reconstruct Master Domain for Data Center Default. >2014-08-03 11:04:45,542 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-44) [5f00c7bf] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue >2014-08-03 11:04:45,544 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-6) starting processDomainRecovery for domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 >2014-08-03 11:04:45,551 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-23) [1802a60f] Failed to refresh VDS , vds = 17f7fe56-5f32-40ef-b098-611b84484e0b : rack, VDS Network Error, continuing. >java.net.ConnectException: Connection refused >2014-08-03 11:04:45,613 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-6) Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 was reported by all hosts in status UP as problematic. Not moving the domain to NonOperational because it is being reconstructed now. >2014-08-03 11:04:45,632 WARN [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-6) [4b3ec461] CanDoAction of action ReconstructMasterDomain failed. Reasons:VAR__ACTION__RECONSTRUCT_MASTER,VAR__TYPE__STORAGE__DOMAIN,ACTION_TYPE_FAILED_NO_VDS_IN_POOL >2014-08-03 11:04:45,639 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-6) [4b3ec461] Removing vds [] from the domain in maintenance cache >2014-08-03 11:04:45,640 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-6) [4b3ec461] Removing host(s) [] from hosts unseen domain report cache >2014-08-03 11:04:45,641 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-44) [5f00c7bf] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue >2014-08-03 11:04:45,643 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-5) starting processDomainRecovery for domain 44559ec0-6bef-491c-bbd6-e67278f37987:export >2014-08-03 11:04:45,648 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-5) Domain 44559ec0-6bef-491c-bbd6-e67278f37987:export was reported by all hosts in status UP as problematic. Moving the domain to NonOperational. >2014-08-03 11:04:45,671 INFO [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-5) [6f57b46c] Lock Acquired to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE >, sharedLocks= ] >2014-08-03 11:04:45,679 INFO [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-5) [6f57b46c] Running command: DeactivateStorageDomainCommand internal: true. Entities affected : ID: 44559ec0-6bef-491c-bbd6-e67278f37987 Type: Storage >2014-08-03 11:04:45,687 INFO [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-5) [6f57b46c] Lock freed to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE >, sharedLocks= ] >2014-08-03 11:04:45,700 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-5) [6f57b46c] Correlation ID: 6f57b46c, Job ID: c59687b5-9359-421d-92bd-1cc5b1fb4a57, Call Stack: null, Custom Event ID: -1, Message: Storage Domain export (Data Center Default) was deactivated by system because it's not visible by any of the hosts. >2014-08-03 11:04:45,708 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-5) [6f57b46c] Removing vds [] from the domain in maintenance cache >2014-08-03 11:04:45,708 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-5) [6f57b46c] Removing host(s) [] from hosts unseen domain report cache >2014-08-03 11:04:48,845 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-43) START, GetHardwareInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]), log id: 584b8f03 >2014-08-03 11:04:48,901 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-43) FINISH, GetHardwareInfoVDSCommand, log id: 584b8f03 >2014-08-03 11:04:48,926 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-43) [7c2b8d14] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS >2014-08-03 11:04:48,976 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-43) [2bc952cd] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-08-03 11:04:48,984 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-43) [34c7414c] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-08-03 11:04:48,997 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-43) [34c7414c] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, storageType = NFS, connectionList = [{ id: 05047c9a-152e-4ef2-b118-4a1753c1eae0, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35export, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 7ab136d4 >2014-08-03 11:04:49,057 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-43) [34c7414c] FINISH, ConnectStorageServerVDSCommand, return: {5083f370-af09-457b-9b73-3440e8b65a07=0, 05047c9a-152e-4ef2-b118-4a1753c1eae0=0}, log id: 7ab136d4 >2014-08-03 11:04:49,058 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-43) [34c7414c] Host rack storage connection was succeeded >2014-08-03 11:04:49,066 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-24) START, ConnectStoragePoolVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, masterVersion = 1), log id: 3eaf384d >2014-08-03 11:04:55,507 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-100) [3b9a9fa6] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Inactive to Unknown, reason : no reporting hosts >2014-08-03 11:04:55,511 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-100) [3b9a9fa6] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Inactive to Unknown, reason : no reporting hosts >2014-08-03 11:05:49,536 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-24) FINISH, ConnectStoragePoolVDSCommand, log id: 3eaf384d >2014-08-03 11:05:49,576 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-43) [34c7414c] START, SetMOMPolicyParametersVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 77df883c >2014-08-03 11:05:49,669 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-43) [34c7414c] FINISH, SetMOMPolicyParametersVDSCommand, log id: 77df883c >2014-08-03 11:05:49,677 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-43) [34c7414c] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host rack was set to Up. >2014-08-03 11:05:49,683 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-43) [34c7414c] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host rack. >2014-08-03 11:05:49,699 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-43) [56fe64d8] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS >2014-08-03 11:05:55,656 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-3) hostFromVds::selectedVds - rack, spmStatus Free, storage pool Default >2014-08-03 11:05:55,663 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-3) starting spm on vds rack, storage pool Default, prevId -1, LVER -1 >2014-08-03 11:05:55,665 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-3) START, SpmStartVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, prevId=-1, prevLVER=-1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 78f11151 >2014-08-03 11:05:55,713 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-3) spmStart polling started: taskId = b00c45e9-c194-40dd-aca3-461bc8ed5ab7 >2014-08-03 11:05:56,720 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-3) spmStart polling ended: taskId = b00c45e9-c194-40dd-aca3-461bc8ed5ab7 task status = finished >2014-08-03 11:05:56,778 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-3) spmStart polling ended, spm status: SPM >2014-08-03 11:05:56,780 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-3) START, HSMClearTaskVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, taskId=b00c45e9-c194-40dd-aca3-461bc8ed5ab7), log id: 73344168 >2014-08-03 11:05:56,826 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-3) FINISH, HSMClearTaskVDSCommand, log id: 73344168 >2014-08-03 11:05:56,826 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-3) FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@5d023b8a, log id: 78f11151 >2014-08-03 11:05:56,830 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-3) Initialize Irs proxy from vds: 10.34.62.208 >2014-08-03 11:05:56,836 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-3) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208). >2014-08-03 11:05:56,847 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-11) START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 6594af2f >2014-08-03 11:05:56,876 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-11) -- executeIrsBrokerCommand: Attempting on storage pool 00000002-0002-0002-0002-0000000001d9 >2014-08-03 11:05:56,878 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-11) START, HSMGetAllTasksInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 659a1882 >2014-08-03 11:05:56,884 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-11) FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 659a1882 >2014-08-03 11:05:56,885 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-11) FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 6594af2f >2014-08-03 11:05:56,886 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (org.ovirt.thread.pool-8-thread-11) Discovered no tasks on Storage Pool Default >2014-08-03 11:23:55,817 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-37) Principal admin::internal synchronized >2014-08-03 11:24:24,005 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-29) [327b509d] Attempting to update VMs/Templates Ovf. >2014-08-03 12:23:55,822 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-13) [519c108f] Principal admin::internal synchronized >2014-08-03 12:24:24,025 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-11) [aeb0141] Attempting to update VMs/Templates Ovf. >2014-08-03 12:53:24,194 WARN [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-8-thread-44) Executing a command: java.util.concurrent.FutureTask , but note that there are 1 tasks in the queue. >2014-08-03 13:23:55,830 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-20) [367a2664] Principal admin::internal synchronized >2014-08-03 13:24:24,052 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-95) [4c7ac6a1] Attempting to update VMs/Templates Ovf. >2014-08-03 14:05:44,214 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-28) domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 in problem. vds: rack >2014-08-03 14:05:44,216 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-28) domain 44559ec0-6bef-491c-bbd6-e67278f37987:export in problem. vds: rack >2014-08-03 14:07:13,583 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-42) [161c5527] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2014-08-03 14:07:13,587 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-42) [708ffdc9] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-08-03 14:07:13,590 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-42) [708ffdc9] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Active to Unknown, reason : null >2014-08-03 14:07:13,592 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-42) [708ffdc9] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Active to Unknown, reason : null >2014-08-03 14:07:13,604 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-42) [708ffdc9] Correlation ID: 708ffdc9, Call Stack: null, Custom Event ID: -1, Message: Invalid status on Data Center Default. Setting Data Center status to Non Responsive (On host rack, Error: Cannot access master storage domain). >2014-08-03 14:09:13,637 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-42) [708ffdc9] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2014-08-03 14:09:13,640 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-42) [708ffdc9] hostFromVds::selectedVds - rack, spmStatus returned null! >2014-08-03 14:09:13,641 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-42) [708ffdc9] IrsBroker::Failed::GetStoragePoolInfoVDS >2014-08-03 14:09:13,642 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-42) [708ffdc9] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2014-08-03 14:09:13,685 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-39) [3b915adf] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage >2014-08-03 14:09:13,705 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-39) [3b915adf] START, ResetIrsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, ignoreStopFailed = true), log id: 3c1fa393 >2014-08-03 14:09:13,709 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-39) [3b915adf] START, SpmStopVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9), log id: 695e1ab0 >2014-08-03 14:09:13,716 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-39) [3b915adf] SpmStopVDSCommand::Stopping SPM on vds rack, pool id 00000002-0002-0002-0002-0000000001d9 >2014-08-03 14:10:00,006 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-38) [1b9ef0f] Autorecovering 1 storage domains >2014-08-03 14:10:00,007 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-38) [1b9ef0f] Autorecovering storage domains id: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 >2014-08-03 14:10:00,009 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-38) [4bd2cb12] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage >2014-08-03 14:10:00,011 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-38) [4bd2cb12] ConnectDomainToStorage. Before Connect all hosts to pool. Time:8/3/14 2:10 PM >2014-08-03 14:10:00,020 INFO [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-8-thread-16) [421a0427] Running command: ConnectStorageToVdsCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2014-08-03 14:10:00,024 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-16) [421a0427] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 74dc7bbc >2014-08-03 14:11:13,605 WARN [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-39) [3b915adf] could not stop spm of pool 00000002-0002-0002-0002-0000000001d9 on vds 17f7fe56-5f32-40ef-b098-611b84484e0b - reason: org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcRunTimeException: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException >2014-08-03 14:11:13,610 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ListVDSCommand] (DefaultQuartzScheduler_Worker-98) [5aa74089] Command ListVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-08-03 14:11:13,610 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-39) [3b915adf] FINISH, SpmStopVDSCommand, log id: 695e1ab0 >2014-08-03 14:11:13,614 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-16) [421a0427] Command ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-08-03 14:11:13,613 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-98) [5aa74089] Host rack is not responding. It will stay in Connecting state for a grace period of $80 seconds and after that an attempt to fence the host will be issued. >2014-08-03 14:11:13,618 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-16) [421a0427] FINISH, ConnectStorageServerVDSCommand, log id: 74dc7bbc >2014-08-03 14:11:13,615 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-4) Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: VDSNetworkException: org.apache.commons.httpclient.NoHttpResponseException: The server 10.34.62.208 failed to respond >2014-08-03 14:11:13,623 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-4) hostFromVds::selectedVds - rack, spmStatus returned null! >2014-08-03 14:11:13,623 ERROR [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-8-thread-16) [421a0427] Command org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-08-03 14:11:13,622 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-39) [3b915adf] FINISH, ResetIrsVDSCommand, log id: 3c1fa393 >2014-08-03 14:11:13,619 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-98) [5aa74089] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:406) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.fetchRunningVms(VdsUpdateRunTimeInfo.java:971) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVmStats(VdsUpdateRunTimeInfo.java:920) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:638) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:235) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) [:1.7.0_55] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_55] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_55] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_55] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_55] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_55] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_55] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_55] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy82.list(Unknown Source) > at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source) [:1.7.0_55] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_55] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_55] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_55] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_55] > >2014-08-03 14:11:13,636 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-39) [3b915adf] Correlation ID: 3b915adf, Job ID: 668cbbdf-0b1a-4927-b758-2c8b93beded3, Call Stack: null, Custom Event ID: -1, Message: Failed to Reconstruct Master Domain for Data Center Default. >2014-08-03 14:11:13,631 ERROR [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-8-thread-16) [421a0427] Transaction rolled-back for command: org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand. >2014-08-03 14:11:13,658 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-38) [4bd2cb12] ConnectDomainToStorage. After Connect all hosts to pool. Time:8/3/14 2:11 PM >2014-08-03 14:11:13,659 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-98) [5aa74089] Failed to refresh VDS , vds = 17f7fe56-5f32-40ef-b098-611b84484e0b : rack, VDS Network Error, continuing. >java.net.ConnectException: Connection refused >2014-08-03 14:11:13,658 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-49) [708ffdc9] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue >2014-08-03 14:11:13,663 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-33) starting processDomainRecovery for domain 44559ec0-6bef-491c-bbd6-e67278f37987:export >2014-08-03 14:11:13,724 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-33) Domain 44559ec0-6bef-491c-bbd6-e67278f37987:export was reported by all hosts in status UP as problematic. Moving the domain to NonOperational. >2014-08-03 14:11:13,743 INFO [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-33) [17ef6b50] Lock Acquired to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE >, sharedLocks= ] >2014-08-03 14:11:13,750 INFO [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-33) [17ef6b50] Running command: DeactivateStorageDomainCommand internal: true. Entities affected : ID: 44559ec0-6bef-491c-bbd6-e67278f37987 Type: Storage >2014-08-03 14:11:13,767 INFO [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-33) [17ef6b50] Lock freed to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE >, sharedLocks= ] >2014-08-03 14:11:13,777 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-33) [17ef6b50] Correlation ID: 17ef6b50, Job ID: f8272be9-841f-47fe-9989-00ed56f47cbd, Call Stack: null, Custom Event ID: -1, Message: Storage Domain export (Data Center Default) was deactivated by system because it's not visible by any of the hosts. >2014-08-03 14:11:13,782 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-33) [17ef6b50] Removing vds [] from the domain in maintenance cache >2014-08-03 14:11:13,783 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-33) [17ef6b50] Removing host(s) [] from hosts unseen domain report cache >2014-08-03 14:11:13,785 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-37) starting processDomainRecovery for domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 >2014-08-03 14:11:13,790 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-37) Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 was reported by all hosts in status UP as problematic. Not moving the domain to NonOperational because it is being reconstructed now. >2014-08-03 14:11:13,807 WARN [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-37) [66828990] CanDoAction of action ReconstructMasterDomain failed. Reasons:VAR__ACTION__RECONSTRUCT_MASTER,VAR__TYPE__STORAGE__DOMAIN,ACTION_TYPE_FAILED_NO_VDS_IN_POOL >2014-08-03 14:11:13,814 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-37) [66828990] Removing vds [] from the domain in maintenance cache >2014-08-03 14:11:13,815 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-37) [66828990] Removing host(s) [] from hosts unseen domain report cache >2014-08-03 14:11:13,816 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-49) [708ffdc9] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue >2014-08-03 14:11:17,172 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-72) START, GetHardwareInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]), log id: 7fc96f76 >2014-08-03 14:11:17,221 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-72) FINISH, GetHardwareInfoVDSCommand, log id: 7fc96f76 >2014-08-03 14:11:17,245 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-72) [2e7ea298] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS >2014-08-03 14:11:17,295 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-72) [5088e5e4] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-08-03 14:11:17,303 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-72) [59c7ae33] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-08-03 14:11:17,315 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-72) [59c7ae33] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, storageType = NFS, connectionList = [{ id: 05047c9a-152e-4ef2-b118-4a1753c1eae0, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35export, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 78297031 >2014-08-03 14:11:17,379 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-72) [59c7ae33] FINISH, ConnectStorageServerVDSCommand, return: {5083f370-af09-457b-9b73-3440e8b65a07=0, 05047c9a-152e-4ef2-b118-4a1753c1eae0=0}, log id: 78297031 >2014-08-03 14:11:17,380 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-72) [59c7ae33] Host rack storage connection was succeeded >2014-08-03 14:11:17,388 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-14) START, ConnectStoragePoolVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, masterVersion = 1), log id: 247bfb93 >2014-08-03 14:11:23,628 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-70) [5f00c7bf] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Inactive to Unknown, reason : no reporting hosts >2014-08-03 14:11:23,632 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-70) [5f00c7bf] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Inactive to Unknown, reason : no reporting hosts >2014-08-03 14:12:17,812 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-14) FINISH, ConnectStoragePoolVDSCommand, log id: 247bfb93 >2014-08-03 14:12:17,826 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-72) [59c7ae33] START, SetMOMPolicyParametersVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 910d1b3 >2014-08-03 14:12:17,919 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-72) [59c7ae33] FINISH, SetMOMPolicyParametersVDSCommand, log id: 910d1b3 >2014-08-03 14:12:17,927 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-72) [59c7ae33] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host rack was set to Up. >2014-08-03 14:12:17,936 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-72) [59c7ae33] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host rack. >2014-08-03 14:12:17,951 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-72) [275ae126] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS >2014-08-03 14:12:23,743 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-50) [16080eb6] hostFromVds::selectedVds - rack, spmStatus Free, storage pool Default >2014-08-03 14:12:23,750 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-50) [16080eb6] starting spm on vds rack, storage pool Default, prevId -1, LVER -1 >2014-08-03 14:12:23,754 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-50) [16080eb6] START, SpmStartVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, prevId=-1, prevLVER=-1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 2730a707 >2014-08-03 14:12:23,804 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-50) [16080eb6] spmStart polling started: taskId = 515c9f8e-e8c2-4f45-bbc1-b9fc292222ce >2014-08-03 14:12:24,809 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-50) [16080eb6] spmStart polling ended: taskId = 515c9f8e-e8c2-4f45-bbc1-b9fc292222ce task status = finished >2014-08-03 14:12:24,865 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-50) [16080eb6] spmStart polling ended, spm status: SPM >2014-08-03 14:12:24,867 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-50) [16080eb6] START, HSMClearTaskVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, taskId=515c9f8e-e8c2-4f45-bbc1-b9fc292222ce), log id: 7ce97c7b >2014-08-03 14:12:24,912 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-50) [16080eb6] FINISH, HSMClearTaskVDSCommand, log id: 7ce97c7b >2014-08-03 14:12:24,914 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-50) [16080eb6] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@a65aa9f, log id: 2730a707 >2014-08-03 14:12:24,918 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-50) [16080eb6] Initialize Irs proxy from vds: 10.34.62.208 >2014-08-03 14:12:24,925 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-50) [16080eb6] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208). >2014-08-03 14:12:24,926 WARN [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-8-thread-3) Executing a command: java.util.concurrent.FutureTask , but note that there are 1 tasks in the queue. >2014-08-03 14:12:24,938 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-3) [16080eb6] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 3a1df1b3 >2014-08-03 14:12:24,965 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-3) [16080eb6] -- executeIrsBrokerCommand: Attempting on storage pool 00000002-0002-0002-0002-0000000001d9 >2014-08-03 14:12:24,967 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-3) [16080eb6] START, HSMGetAllTasksInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 48ebd35a >2014-08-03 14:12:24,973 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-3) [16080eb6] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 48ebd35a >2014-08-03 14:12:24,974 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-3) [16080eb6] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 3a1df1b3 >2014-08-03 14:12:24,975 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (org.ovirt.thread.pool-8-thread-3) [16080eb6] Discovered no tasks on Storage Pool Default >2014-08-03 14:23:55,837 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-86) [6452f878] Principal admin::internal synchronized >2014-08-03 14:24:24,068 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-5) Attempting to update VMs/Templates Ovf. >2014-08-03 15:23:55,843 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-31) Principal admin::internal synchronized >2014-08-03 15:24:24,083 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-15) [39225a1e] Attempting to update VMs/Templates Ovf. >2014-08-03 16:23:55,849 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-7) [42c2440e] Principal admin::internal synchronized >2014-08-03 16:24:24,097 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-17) Attempting to update VMs/Templates Ovf. >2014-08-03 17:12:03,451 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-31) domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 in problem. vds: rack >2014-08-03 17:12:18,572 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-18) domain 44559ec0-6bef-491c-bbd6-e67278f37987:export in problem. vds: rack >2014-08-03 17:13:43,119 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-95) [4c7ac6a1] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2014-08-03 17:13:43,124 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-95) [1abadf13] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-08-03 17:13:43,128 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-95) [1abadf13] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Active to Unknown, reason : null >2014-08-03 17:13:43,131 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-95) [1abadf13] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Active to Unknown, reason : null >2014-08-03 17:13:43,147 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-95) [1abadf13] Correlation ID: 1abadf13, Call Stack: null, Custom Event ID: -1, Message: Invalid status on Data Center Default. Setting Data Center status to Non Responsive (On host rack, Error: Cannot access master storage domain). >2014-08-03 17:15:43,145 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-95) [1abadf13] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2014-08-03 17:15:43,147 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-95) [1abadf13] hostFromVds::selectedVds - rack, spmStatus returned null! >2014-08-03 17:15:43,148 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-95) [1abadf13] IrsBroker::Failed::GetStoragePoolInfoVDS >2014-08-03 17:15:43,149 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-95) [1abadf13] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2014-08-03 17:15:43,185 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-29) [513e5878] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage >2014-08-03 17:15:43,203 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-29) [513e5878] START, ResetIrsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, ignoreStopFailed = true), log id: 3e0dce79 >2014-08-03 17:15:43,206 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-29) [513e5878] START, SpmStopVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9), log id: 55f492d4 >2014-08-03 17:15:43,218 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-29) [513e5878] SpmStopVDSCommand::Stopping SPM on vds rack, pool id 00000002-0002-0002-0002-0000000001d9 >2014-08-03 17:17:43,154 WARN [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-29) [513e5878] could not stop spm of pool 00000002-0002-0002-0002-0000000001d9 on vds 17f7fe56-5f32-40ef-b098-611b84484e0b - reason: org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcRunTimeException: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException >2014-08-03 17:17:43,160 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ListVDSCommand] (DefaultQuartzScheduler_Worker-85) [5e890dae] Command ListVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-08-03 17:17:43,159 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-50) [16080eb6] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: VDSNetworkException: org.apache.commons.httpclient.NoHttpResponseException: The server 10.34.62.208 failed to respond >2014-08-03 17:17:43,162 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-85) [5e890dae] Host rack is not responding. It will stay in Connecting state for a grace period of $80 seconds and after that an attempt to fence the host will be issued. >2014-08-03 17:17:43,160 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-29) [513e5878] FINISH, SpmStopVDSCommand, log id: 55f492d4 >2014-08-03 17:17:43,166 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-50) [16080eb6] hostFromVds::selectedVds - rack, spmStatus returned null! >2014-08-03 17:17:43,168 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-85) [5e890dae] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:406) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.fetchRunningVms(VdsUpdateRunTimeInfo.java:971) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVmStats(VdsUpdateRunTimeInfo.java:920) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:638) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:235) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) [:1.7.0_55] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_55] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_55] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_55] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_55] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_55] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_55] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_55] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy82.list(Unknown Source) > at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source) [:1.7.0_55] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_55] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_55] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_55] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_55] > >2014-08-03 17:17:43,176 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-29) [513e5878] FINISH, ResetIrsVDSCommand, log id: 3e0dce79 >2014-08-03 17:17:43,200 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-85) [5e890dae] Failed to refresh VDS , vds = 17f7fe56-5f32-40ef-b098-611b84484e0b : rack, VDS Network Error, continuing. >java.net.ConnectException: Connection refused >2014-08-03 17:17:43,209 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-29) [513e5878] Correlation ID: 513e5878, Job ID: bb490939-775a-45b0-a39f-6a991a569306, Call Stack: null, Custom Event ID: -1, Message: Failed to Reconstruct Master Domain for Data Center Default. >2014-08-03 17:17:43,217 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-6) [1abadf13] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue >2014-08-03 17:17:43,219 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-33) starting processDomainRecovery for domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 >2014-08-03 17:17:43,279 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-33) Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 was reported by all hosts in status UP as problematic. Not moving the domain to NonOperational because it is being reconstructed now. >2014-08-03 17:17:43,297 WARN [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-33) [78ed91ec] CanDoAction of action ReconstructMasterDomain failed. Reasons:VAR__ACTION__RECONSTRUCT_MASTER,VAR__TYPE__STORAGE__DOMAIN,ACTION_TYPE_FAILED_NO_VDS_IN_POOL >2014-08-03 17:17:43,306 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-33) [78ed91ec] Removing vds [] from the domain in maintenance cache >2014-08-03 17:17:43,307 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-33) [78ed91ec] Removing host(s) [] from hosts unseen domain report cache >2014-08-03 17:17:43,308 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-6) [1abadf13] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue >2014-08-03 17:17:43,310 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-8) starting processDomainRecovery for domain 44559ec0-6bef-491c-bbd6-e67278f37987:export >2014-08-03 17:17:43,315 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-8) Domain 44559ec0-6bef-491c-bbd6-e67278f37987:export was reported by all hosts in status UP as problematic. Moving the domain to NonOperational. >2014-08-03 17:17:43,329 INFO [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-8) [319458de] Lock Acquired to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE >, sharedLocks= ] >2014-08-03 17:17:43,336 INFO [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-8) [319458de] Running command: DeactivateStorageDomainCommand internal: true. Entities affected : ID: 44559ec0-6bef-491c-bbd6-e67278f37987 Type: Storage >2014-08-03 17:17:43,341 INFO [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-8) [319458de] Lock freed to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE >, sharedLocks= ] >2014-08-03 17:17:43,351 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-8) [319458de] Correlation ID: 319458de, Job ID: 9e46c5f7-417b-4e95-981c-62ecab05322b, Call Stack: null, Custom Event ID: -1, Message: Storage Domain export (Data Center Default) was deactivated by system because it's not visible by any of the hosts. >2014-08-03 17:17:43,356 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-8) [319458de] Removing vds [] from the domain in maintenance cache >2014-08-03 17:17:43,357 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-8) [319458de] Removing host(s) [] from hosts unseen domain report cache >2014-08-03 17:17:46,460 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-73) START, GetHardwareInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]), log id: 692d70c1 >2014-08-03 17:17:46,522 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-73) FINISH, GetHardwareInfoVDSCommand, log id: 692d70c1 >2014-08-03 17:17:46,550 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-73) [76a195c5] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS >2014-08-03 17:17:46,599 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-73) [1603d8f1] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-08-03 17:17:46,607 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-73) [7b33c76f] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-08-03 17:17:46,619 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-73) [7b33c76f] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, storageType = NFS, connectionList = [{ id: 05047c9a-152e-4ef2-b118-4a1753c1eae0, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35export, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 3d4a582 >2014-08-03 17:17:46,678 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-73) [7b33c76f] FINISH, ConnectStorageServerVDSCommand, return: {5083f370-af09-457b-9b73-3440e8b65a07=0, 05047c9a-152e-4ef2-b118-4a1753c1eae0=0}, log id: 3d4a582 >2014-08-03 17:17:46,679 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-73) [7b33c76f] Host rack storage connection was succeeded >2014-08-03 17:17:46,687 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-22) START, ConnectStoragePoolVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, masterVersion = 1), log id: 5b56f9b7 >2014-08-03 17:17:53,176 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-96) Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Inactive to Unknown, reason : no reporting hosts >2014-08-03 17:17:53,181 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-96) Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Inactive to Unknown, reason : no reporting hosts >2014-08-03 17:18:47,078 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-22) FINISH, ConnectStoragePoolVDSCommand, log id: 5b56f9b7 >2014-08-03 17:18:47,121 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-73) [7b33c76f] START, SetMOMPolicyParametersVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 23076021 >2014-08-03 17:18:47,213 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-73) [7b33c76f] FINISH, SetMOMPolicyParametersVDSCommand, log id: 23076021 >2014-08-03 17:18:47,221 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-73) [7b33c76f] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host rack was set to Up. >2014-08-03 17:18:47,227 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-73) [7b33c76f] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host rack. >2014-08-03 17:18:47,268 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-73) [5fdab73] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS >2014-08-03 17:18:53,366 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-8) [782c75f] hostFromVds::selectedVds - rack, spmStatus Free, storage pool Default >2014-08-03 17:18:53,371 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-8) [782c75f] starting spm on vds rack, storage pool Default, prevId -1, LVER -1 >2014-08-03 17:18:53,374 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-8) [782c75f] START, SpmStartVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, prevId=-1, prevLVER=-1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 15876e44 >2014-08-03 17:18:53,423 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-8) [782c75f] spmStart polling started: taskId = e0001b88-a74f-4c57-92c8-3f51c0067d16 >2014-08-03 17:18:54,429 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-8) [782c75f] spmStart polling ended: taskId = e0001b88-a74f-4c57-92c8-3f51c0067d16 task status = finished >2014-08-03 17:18:54,485 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-8) [782c75f] spmStart polling ended, spm status: SPM >2014-08-03 17:18:54,488 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-8) [782c75f] START, HSMClearTaskVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, taskId=e0001b88-a74f-4c57-92c8-3f51c0067d16), log id: 3002d464 >2014-08-03 17:18:54,534 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-8) [782c75f] FINISH, HSMClearTaskVDSCommand, log id: 3002d464 >2014-08-03 17:18:54,536 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-8) [782c75f] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@6b24b935, log id: 15876e44 >2014-08-03 17:18:54,543 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-8) [782c75f] Initialize Irs proxy from vds: 10.34.62.208 >2014-08-03 17:18:54,552 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-8) [782c75f] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208). >2014-08-03 17:18:54,555 WARN [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-8-thread-3) Executing a command: java.util.concurrent.FutureTask , but note that there are 1 tasks in the queue. >2014-08-03 17:18:54,567 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-3) [782c75f] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 23e4f08b >2014-08-03 17:18:54,596 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-3) [782c75f] -- executeIrsBrokerCommand: Attempting on storage pool 00000002-0002-0002-0002-0000000001d9 >2014-08-03 17:18:54,598 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-3) [782c75f] START, HSMGetAllTasksInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 5736c62f >2014-08-03 17:18:54,604 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-3) [782c75f] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 5736c62f >2014-08-03 17:18:54,605 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-3) [782c75f] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 23e4f08b >2014-08-03 17:18:54,605 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (org.ovirt.thread.pool-8-thread-3) [782c75f] Discovered no tasks on Storage Pool Default >2014-08-03 17:23:55,855 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-90) Principal admin::internal synchronized >2014-08-03 17:24:24,130 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-60) [1cb6c42c] Attempting to update VMs/Templates Ovf. >2014-08-03 18:23:55,862 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-15) [39225a1e] Principal admin::internal synchronized >2014-08-03 18:24:24,145 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-45) [22578e65] Attempting to update VMs/Templates Ovf. >2014-08-03 19:23:55,869 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-48) [14cac44e] Principal admin::internal synchronized >2014-08-03 19:24:24,160 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-60) [1cb6c42c] Attempting to update VMs/Templates Ovf. >2014-08-03 20:18:39,494 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-4) domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 in problem. vds: rack >2014-08-03 20:18:39,496 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-4) domain 44559ec0-6bef-491c-bbd6-e67278f37987:export in problem. vds: rack >2014-08-03 20:20:14,597 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-85) [5e890dae] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2014-08-03 20:20:14,601 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-85) [3e29dd44] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-08-03 20:20:14,605 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-85) [3e29dd44] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Active to Unknown, reason : null >2014-08-03 20:20:14,607 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-85) [3e29dd44] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Active to Unknown, reason : null >2014-08-03 20:20:14,619 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-85) [3e29dd44] Correlation ID: 3e29dd44, Call Stack: null, Custom Event ID: -1, Message: Invalid status on Data Center Default. Setting Data Center status to Non Responsive (On host rack, Error: Cannot access master storage domain). >2014-08-03 20:22:14,649 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-85) [3e29dd44] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2014-08-03 20:22:14,653 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-85) [3e29dd44] hostFromVds::selectedVds - rack, spmStatus returned null! >2014-08-03 20:22:14,654 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-85) [3e29dd44] IrsBroker::Failed::GetStoragePoolInfoVDS >2014-08-03 20:22:14,655 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-85) [3e29dd44] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2014-08-03 20:22:14,691 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-15) [1b05141c] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage >2014-08-03 20:22:14,712 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-15) [1b05141c] START, ResetIrsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, ignoreStopFailed = true), log id: 3b956c06 >2014-08-03 20:22:14,717 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-15) [1b05141c] START, SpmStopVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9), log id: 1f25fce3 >2014-08-03 20:22:14,722 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-15) [1b05141c] SpmStopVDSCommand::Stopping SPM on vds rack, pool id 00000002-0002-0002-0002-0000000001d9 >2014-08-03 20:23:55,875 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-31) Principal admin::internal synchronized >2014-08-03 20:24:14,621 WARN [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-15) [1b05141c] could not stop spm of pool 00000002-0002-0002-0002-0000000001d9 on vds 17f7fe56-5f32-40ef-b098-611b84484e0b - reason: org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcRunTimeException: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException >2014-08-03 20:24:14,624 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-15) [1b05141c] FINISH, SpmStopVDSCommand, log id: 1f25fce3 >2014-08-03 20:24:14,623 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetStatsVDSCommand] (DefaultQuartzScheduler_Worker-18) [5497a121] Command GetStatsVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-08-03 20:24:14,622 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-38) [4bd2cb12] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: VDSNetworkException: org.apache.commons.httpclient.NoHttpResponseException: The server 10.34.62.208 failed to respond >2014-08-03 20:24:14,630 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-18) [5497a121] Host rack is not responding. It will stay in Connecting state for a grace period of $80 seconds and after that an attempt to fence the host will be issued. >2014-08-03 20:24:14,631 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-38) [4bd2cb12] hostFromVds::selectedVds - rack, spmStatus returned null! >2014-08-03 20:24:14,633 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-15) [1b05141c] FINISH, ResetIrsVDSCommand, log id: 3b956c06 >2014-08-03 20:24:14,633 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-18) [5497a121] vds::refreshVdsStats Failed getVdsStats, vds = 17f7fe56-5f32-40ef-b098-611b84484e0b : rack, error = VDSNetworkException: java.net.ConnectException: Connection refused >2014-08-03 20:24:14,636 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-18) [5497a121] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsStats(VdsUpdateRunTimeInfo.java:669) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:623) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:235) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) [:1.7.0_55] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_55] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_55] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_55] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_55] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_55] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_55] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_55] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy82.getVdsStats(Unknown Source) > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_55] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_55] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_55] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_55] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_55] > >2014-08-03 20:24:14,643 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-15) [1b05141c] Correlation ID: 1b05141c, Job ID: 3489d1c6-b451-41d5-a2e1-c407c3516776, Call Stack: null, Custom Event ID: -1, Message: Failed to Reconstruct Master Domain for Data Center Default. >2014-08-03 20:24:14,664 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-45) [3e29dd44] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue >2014-08-03 20:24:14,665 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-18) [5497a121] Failed to refresh VDS , vds = 17f7fe56-5f32-40ef-b098-611b84484e0b : rack, VDS Network Error, continuing. >java.net.ConnectException: Connection refused >2014-08-03 20:24:14,666 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-29) starting processDomainRecovery for domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 >2014-08-03 20:24:14,738 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-29) Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 was reported by all hosts in status UP as problematic. Not moving the domain to NonOperational because it is being reconstructed now. >2014-08-03 20:24:14,756 WARN [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-29) [80dee22] CanDoAction of action ReconstructMasterDomain failed. Reasons:VAR__ACTION__RECONSTRUCT_MASTER,VAR__TYPE__STORAGE__DOMAIN,ACTION_TYPE_FAILED_NO_VDS_IN_POOL >2014-08-03 20:24:14,763 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-29) [80dee22] Removing vds [] from the domain in maintenance cache >2014-08-03 20:24:14,765 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-29) [80dee22] Removing host(s) [] from hosts unseen domain report cache >2014-08-03 20:24:14,767 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-45) [3e29dd44] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue >2014-08-03 20:24:14,770 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-34) starting processDomainRecovery for domain 44559ec0-6bef-491c-bbd6-e67278f37987:export >2014-08-03 20:24:14,776 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-34) Domain 44559ec0-6bef-491c-bbd6-e67278f37987:export was reported by all hosts in status UP as problematic. Moving the domain to NonOperational. >2014-08-03 20:24:14,792 INFO [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-34) [6054abd1] Lock Acquired to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE >, sharedLocks= ] >2014-08-03 20:24:14,798 INFO [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-34) [6054abd1] Running command: DeactivateStorageDomainCommand internal: true. Entities affected : ID: 44559ec0-6bef-491c-bbd6-e67278f37987 Type: Storage >2014-08-03 20:24:14,804 INFO [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-34) [6054abd1] Lock freed to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE >, sharedLocks= ] >2014-08-03 20:24:14,813 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-34) [6054abd1] Correlation ID: 6054abd1, Job ID: 1358fee6-c04d-4e99-9b94-0ebe8c5d5189, Call Stack: null, Custom Event ID: -1, Message: Storage Domain export (Data Center Default) was deactivated by system because it's not visible by any of the hosts. >2014-08-03 20:24:14,818 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-34) [6054abd1] Removing vds [] from the domain in maintenance cache >2014-08-03 20:24:14,819 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-34) [6054abd1] Removing host(s) [] from hosts unseen domain report cache >2014-08-03 20:24:17,921 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-30) [dd7f8b1] START, GetHardwareInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]), log id: 157ad5c1 >2014-08-03 20:24:17,975 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-30) [dd7f8b1] FINISH, GetHardwareInfoVDSCommand, log id: 157ad5c1 >2014-08-03 20:24:18,005 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-30) [78587bc7] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS >2014-08-03 20:24:18,057 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-30) [139f8621] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-08-03 20:24:18,064 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-30) [66f70bde] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-08-03 20:24:18,070 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-30) [66f70bde] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, storageType = NFS, connectionList = [{ id: 05047c9a-152e-4ef2-b118-4a1753c1eae0, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35export, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 5845eeec >2014-08-03 20:24:18,130 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-30) [66f70bde] FINISH, ConnectStorageServerVDSCommand, return: {5083f370-af09-457b-9b73-3440e8b65a07=0, 05047c9a-152e-4ef2-b118-4a1753c1eae0=0}, log id: 5845eeec >2014-08-03 20:24:18,131 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-30) [66f70bde] Host rack storage connection was succeeded >2014-08-03 20:24:18,137 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-41) START, ConnectStoragePoolVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, masterVersion = 1), log id: 477818a3 >2014-08-03 20:24:24,182 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-53) [3c2dfb77] Attempting to update VMs/Templates Ovf. >2014-08-03 20:24:24,638 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-1) [795e8140] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Inactive to Unknown, reason : no reporting hosts >2014-08-03 20:24:24,644 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-1) [795e8140] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Inactive to Unknown, reason : no reporting hosts >2014-08-03 20:25:18,565 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-41) FINISH, ConnectStoragePoolVDSCommand, log id: 477818a3 >2014-08-03 20:25:18,578 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-30) [66f70bde] START, SetMOMPolicyParametersVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 4790eeda >2014-08-03 20:25:18,697 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-30) [66f70bde] FINISH, SetMOMPolicyParametersVDSCommand, log id: 4790eeda >2014-08-03 20:25:18,705 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-30) [66f70bde] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host rack was set to Up. >2014-08-03 20:25:18,711 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-30) [66f70bde] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host rack. >2014-08-03 20:25:18,730 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-30) [3adcf236] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS >2014-08-03 20:25:24,808 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-22) [7e0971f4] hostFromVds::selectedVds - rack, spmStatus Free, storage pool Default >2014-08-03 20:25:24,814 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-22) [7e0971f4] starting spm on vds rack, storage pool Default, prevId -1, LVER -1 >2014-08-03 20:25:24,817 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-22) [7e0971f4] START, SpmStartVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, prevId=-1, prevLVER=-1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 177fac0c >2014-08-03 20:25:24,866 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-22) [7e0971f4] spmStart polling started: taskId = 6a664af8-94a4-46f6-8bb6-8285a696ad10 >2014-08-03 20:25:25,872 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-22) [7e0971f4] spmStart polling ended: taskId = 6a664af8-94a4-46f6-8bb6-8285a696ad10 task status = finished >2014-08-03 20:25:25,928 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-22) [7e0971f4] spmStart polling ended, spm status: SPM >2014-08-03 20:25:25,930 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-22) [7e0971f4] START, HSMClearTaskVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, taskId=6a664af8-94a4-46f6-8bb6-8285a696ad10), log id: 33bb9374 >2014-08-03 20:25:25,976 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-22) [7e0971f4] FINISH, HSMClearTaskVDSCommand, log id: 33bb9374 >2014-08-03 20:25:25,976 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-22) [7e0971f4] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@686dcc01, log id: 177fac0c >2014-08-03 20:25:25,980 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-22) [7e0971f4] Initialize Irs proxy from vds: 10.34.62.208 >2014-08-03 20:25:25,988 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-22) [7e0971f4] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208). >2014-08-03 20:25:25,989 WARN [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-8-thread-27) Executing a command: java.util.concurrent.FutureTask , but note that there are 1 tasks in the queue. >2014-08-03 20:25:25,998 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-27) [7e0971f4] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 20b30b90 >2014-08-03 20:25:26,040 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-27) [7e0971f4] -- executeIrsBrokerCommand: Attempting on storage pool 00000002-0002-0002-0002-0000000001d9 >2014-08-03 20:25:26,042 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-27) [7e0971f4] START, HSMGetAllTasksInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 351b6555 >2014-08-03 20:25:26,047 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-27) [7e0971f4] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 351b6555 >2014-08-03 20:25:26,048 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-27) [7e0971f4] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 20b30b90 >2014-08-03 20:25:26,049 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (org.ovirt.thread.pool-8-thread-27) [7e0971f4] Discovered no tasks on Storage Pool Default >2014-08-03 20:27:16,796 WARN [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-8-thread-38) Executing a command: java.util.concurrent.FutureTask , but note that there are 1 tasks in the queue. >2014-08-03 20:27:16,797 WARN [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-8-thread-31) Executing a command: java.util.concurrent.FutureTask , but note that there are 1 tasks in the queue. >2014-08-03 21:23:55,881 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-45) [22578e65] Principal admin::internal synchronized >2014-08-03 21:24:24,185 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-37) Attempting to update VMs/Templates Ovf. >2014-08-03 22:23:55,886 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-65) Principal admin::internal synchronized >2014-08-03 22:24:24,203 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-7) [42c2440e] Attempting to update VMs/Templates Ovf. >2014-08-03 23:23:55,893 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-24) [61a1f9b5] Principal admin::internal synchronized >2014-08-03 23:24:24,218 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-21) [81f884c] Attempting to update VMs/Templates Ovf. >2014-08-03 23:25:13,035 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-44) domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 in problem. vds: rack >2014-08-03 23:25:13,037 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-44) domain 44559ec0-6bef-491c-bbd6-e67278f37987:export in problem. vds: rack >2014-08-03 23:26:49,201 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-33) [561d02fb] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2014-08-03 23:26:49,209 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-33) [6bf204c4] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-08-03 23:26:49,214 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-33) [6bf204c4] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Active to Unknown, reason : null >2014-08-03 23:26:49,219 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-33) [6bf204c4] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Active to Unknown, reason : null >2014-08-03 23:26:49,234 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-33) [6bf204c4] Correlation ID: 6bf204c4, Call Stack: null, Custom Event ID: -1, Message: Invalid status on Data Center Default. Setting Data Center status to Non Responsive (On host rack, Error: Cannot access master storage domain). >2014-08-03 23:28:49,248 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-33) [6bf204c4] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2014-08-03 23:28:49,250 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-33) [6bf204c4] hostFromVds::selectedVds - rack, spmStatus returned null! >2014-08-03 23:28:49,252 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-33) [6bf204c4] IrsBroker::Failed::GetStoragePoolInfoVDS >2014-08-03 23:28:49,253 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-33) [6bf204c4] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2014-08-03 23:28:49,305 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-5) [58abd855] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage >2014-08-03 23:28:49,331 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-5) [58abd855] START, ResetIrsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, ignoreStopFailed = true), log id: 310d312f >2014-08-03 23:28:49,335 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-5) [58abd855] START, SpmStopVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9), log id: 2d32a150 >2014-08-03 23:28:49,341 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-5) [58abd855] SpmStopVDSCommand::Stopping SPM on vds rack, pool id 00000002-0002-0002-0002-0000000001d9 >2014-08-03 23:30:00,005 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-77) [2e3238db] Autorecovering 1 storage domains >2014-08-03 23:30:00,007 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-77) [2e3238db] Autorecovering storage domains id: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 >2014-08-03 23:30:00,010 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-77) [1f7efd53] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage >2014-08-03 23:30:00,011 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-77) [1f7efd53] ConnectDomainToStorage. Before Connect all hosts to pool. Time:8/3/14 11:30 PM >2014-08-03 23:30:00,020 INFO [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-8-thread-23) [26957001] Running command: ConnectStorageToVdsCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2014-08-03 23:30:00,025 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-23) [26957001] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 33b2f788 >2014-08-03 23:30:49,218 WARN [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-5) [58abd855] could not stop spm of pool 00000002-0002-0002-0002-0000000001d9 on vds 17f7fe56-5f32-40ef-b098-611b84484e0b - reason: org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcRunTimeException: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException >2014-08-03 23:30:49,223 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-78) Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: VDSNetworkException: org.apache.commons.httpclient.NoHttpResponseException: The server 10.34.62.208 failed to respond >2014-08-03 23:30:49,222 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-23) [26957001] Command ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-08-03 23:30:49,231 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-23) [26957001] FINISH, ConnectStorageServerVDSCommand, log id: 33b2f788 >2014-08-03 23:30:49,229 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetStatsVDSCommand] (DefaultQuartzScheduler_Worker-56) [65d9de58] Command GetStatsVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-08-03 23:30:49,228 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-5) [58abd855] FINISH, SpmStopVDSCommand, log id: 2d32a150 >2014-08-03 23:30:49,228 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-78) hostFromVds::selectedVds - rack, spmStatus returned null! >2014-08-03 23:30:49,236 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-56) [65d9de58] Host rack is not responding. It will stay in Connecting state for a grace period of $80 seconds and after that an attempt to fence the host will be issued. >2014-08-03 23:30:49,238 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-56) [65d9de58] vds::refreshVdsStats Failed getVdsStats, vds = 17f7fe56-5f32-40ef-b098-611b84484e0b : rack, error = VDSNetworkException: java.net.ConnectException: Connection refused >2014-08-03 23:30:49,233 ERROR [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-8-thread-23) [26957001] Command org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-08-03 23:30:49,239 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-56) [65d9de58] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsStats(VdsUpdateRunTimeInfo.java:669) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:623) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:235) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) [:1.7.0_55] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_55] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_55] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_55] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_55] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_55] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_55] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_55] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy82.getVdsStats(Unknown Source) > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_55] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_55] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_55] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_55] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_55] > >2014-08-03 23:30:49,245 ERROR [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-8-thread-23) [26957001] Transaction rolled-back for command: org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand. >2014-08-03 23:30:49,243 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-5) [58abd855] FINISH, ResetIrsVDSCommand, log id: 310d312f >2014-08-03 23:30:49,285 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-77) [1f7efd53] ConnectDomainToStorage. After Connect all hosts to pool. Time:8/3/14 11:30 PM >2014-08-03 23:30:49,288 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-56) [65d9de58] Failed to refresh VDS , vds = 17f7fe56-5f32-40ef-b098-611b84484e0b : rack, VDS Network Error, continuing. >java.net.ConnectException: Connection refused >2014-08-03 23:30:49,295 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-5) [58abd855] Correlation ID: 58abd855, Job ID: 2a69ce69-cc51-4748-a61d-3aaa4e35b40b, Call Stack: null, Custom Event ID: -1, Message: Failed to Reconstruct Master Domain for Data Center Default. >2014-08-03 23:30:49,303 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-33) [6bf204c4] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue >2014-08-03 23:30:49,305 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-14) starting processDomainRecovery for domain 44559ec0-6bef-491c-bbd6-e67278f37987:export >2014-08-03 23:30:49,371 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-14) Domain 44559ec0-6bef-491c-bbd6-e67278f37987:export was reported by all hosts in status UP as problematic. Moving the domain to NonOperational. >2014-08-03 23:30:49,386 INFO [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-14) [1879c3dc] Lock Acquired to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE >, sharedLocks= ] >2014-08-03 23:30:49,393 INFO [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-14) [1879c3dc] Running command: DeactivateStorageDomainCommand internal: true. Entities affected : ID: 44559ec0-6bef-491c-bbd6-e67278f37987 Type: Storage >2014-08-03 23:30:49,398 INFO [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-14) [1879c3dc] Lock freed to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE >, sharedLocks= ] >2014-08-03 23:30:49,409 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-14) [1879c3dc] Correlation ID: 1879c3dc, Job ID: 60a48859-b76f-4664-a92c-078104fbf974, Call Stack: null, Custom Event ID: -1, Message: Storage Domain export (Data Center Default) was deactivated by system because it's not visible by any of the hosts. >2014-08-03 23:30:49,416 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-14) [1879c3dc] Removing vds [] from the domain in maintenance cache >2014-08-03 23:30:49,417 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-14) [1879c3dc] Removing host(s) [] from hosts unseen domain report cache >2014-08-03 23:30:49,419 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-9) starting processDomainRecovery for domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 >2014-08-03 23:30:49,424 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-9) Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 was reported by all hosts in status UP as problematic. Not moving the domain to NonOperational because it is being reconstructed now. >2014-08-03 23:30:49,441 WARN [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-9) [60d6f2da] CanDoAction of action ReconstructMasterDomain failed. Reasons:VAR__ACTION__RECONSTRUCT_MASTER,VAR__TYPE__STORAGE__DOMAIN,ACTION_TYPE_FAILED_NO_VDS_IN_POOL >2014-08-03 23:30:49,448 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-9) [60d6f2da] Removing vds [] from the domain in maintenance cache >2014-08-03 23:30:49,449 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-9) [60d6f2da] Removing host(s) [] from hosts unseen domain report cache >2014-08-03 23:30:49,449 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-33) [6bf204c4] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue >2014-08-03 23:30:52,362 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-85) [3e29dd44] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@6828084c >2014-08-03 23:30:52,364 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-85) [3e29dd44] HostName = rack >2014-08-03 23:30:52,365 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-85) [3e29dd44] Command GetCapabilitiesVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]) execution failed. Exception: VDSRecoveringException: Recovering from crash or Initializing >2014-08-03 23:30:52,377 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-85) [3e29dd44] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSRecoveringException: Recovering from crash or Initializing > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:16) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:566) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:235) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Host rack is initializing. Message: Recovering from crash or Initializing >2014-08-03 23:30:52,386 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-85) [3e29dd44] Failed to refresh VDS , vds = 17f7fe56-5f32-40ef-b098-611b84484e0b : rack, error = Recovering from crash or Initializing, continuing. >2014-08-03 23:30:55,619 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-73) [5fdab73] START, GetHardwareInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]), log id: ffaf88c >2014-08-03 23:30:55,671 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-73) [5fdab73] FINISH, GetHardwareInfoVDSCommand, log id: ffaf88c >2014-08-03 23:30:55,698 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-73) [728a99d4] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS >2014-08-03 23:30:55,747 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-73) [728a99d4] Cancelling the recovery from crash timer for VDS 17f7fe56-5f32-40ef-b098-611b84484e0b because vds started initializing >2014-08-03 23:30:55,749 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-73) [3fb3de80] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-08-03 23:30:55,758 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-73) [571f42d] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-08-03 23:30:55,771 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-73) [571f42d] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, storageType = NFS, connectionList = [{ id: 05047c9a-152e-4ef2-b118-4a1753c1eae0, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35export, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 147481ae >2014-08-03 23:30:55,836 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-73) [571f42d] FINISH, ConnectStorageServerVDSCommand, return: {5083f370-af09-457b-9b73-3440e8b65a07=0, 05047c9a-152e-4ef2-b118-4a1753c1eae0=0}, log id: 147481ae >2014-08-03 23:30:55,837 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-73) [571f42d] Host rack storage connection was succeeded >2014-08-03 23:30:55,845 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-2) START, ConnectStoragePoolVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, masterVersion = 1), log id: 21582090 >2014-08-03 23:30:59,244 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-49) [72c79e4e] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Inactive to Unknown, reason : no reporting hosts >2014-08-03 23:30:59,248 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-49) [72c79e4e] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Inactive to Unknown, reason : no reporting hosts >2014-08-03 23:31:56,816 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-2) FINISH, ConnectStoragePoolVDSCommand, log id: 21582090 >2014-08-03 23:31:56,830 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-73) [571f42d] START, SetMOMPolicyParametersVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: b1fe2a2 >2014-08-03 23:31:56,924 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-73) [571f42d] FINISH, SetMOMPolicyParametersVDSCommand, log id: b1fe2a2 >2014-08-03 23:31:56,934 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-73) [571f42d] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host rack was set to Up. >2014-08-03 23:31:56,941 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-73) [571f42d] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host rack. >2014-08-03 23:31:56,958 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-73) [55405a19] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS >2014-08-03 23:31:59,369 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-51) [4760e42] hostFromVds::selectedVds - rack, spmStatus Free, storage pool Default >2014-08-03 23:31:59,374 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-51) [4760e42] starting spm on vds rack, storage pool Default, prevId -1, LVER -1 >2014-08-03 23:31:59,377 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-51) [4760e42] START, SpmStartVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, prevId=-1, prevLVER=-1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 40cfa0a4 >2014-08-03 23:31:59,425 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-51) [4760e42] spmStart polling started: taskId = da4e92a9-37ec-46cf-bf9d-731665d0aec0 >2014-08-03 23:32:00,431 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-51) [4760e42] spmStart polling ended: taskId = da4e92a9-37ec-46cf-bf9d-731665d0aec0 task status = finished >2014-08-03 23:32:00,497 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-51) [4760e42] spmStart polling ended, spm status: SPM >2014-08-03 23:32:00,500 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-51) [4760e42] START, HSMClearTaskVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, taskId=da4e92a9-37ec-46cf-bf9d-731665d0aec0), log id: 7b314fd5 >2014-08-03 23:32:00,546 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-51) [4760e42] FINISH, HSMClearTaskVDSCommand, log id: 7b314fd5 >2014-08-03 23:32:00,547 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-51) [4760e42] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@47478612, log id: 40cfa0a4 >2014-08-03 23:32:00,550 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-51) [4760e42] Initialize Irs proxy from vds: 10.34.62.208 >2014-08-03 23:32:00,557 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-51) [4760e42] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208). >2014-08-03 23:32:00,559 WARN [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-8-thread-11) Executing a command: java.util.concurrent.FutureTask , but note that there are 1 tasks in the queue. >2014-08-03 23:32:00,571 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-11) [4760e42] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 4e41b38c >2014-08-03 23:32:00,615 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-11) [4760e42] -- executeIrsBrokerCommand: Attempting on storage pool 00000002-0002-0002-0002-0000000001d9 >2014-08-03 23:32:00,617 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-11) [4760e42] START, HSMGetAllTasksInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 66a4eee0 >2014-08-03 23:32:00,623 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-11) [4760e42] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 66a4eee0 >2014-08-03 23:32:00,624 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-11) [4760e42] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 4e41b38c >2014-08-03 23:32:00,625 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (org.ovirt.thread.pool-8-thread-11) [4760e42] Discovered no tasks on Storage Pool Default >2014-08-04 00:23:55,906 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-35) [5b4b3aec] Principal admin::internal synchronized >2014-08-04 00:24:24,234 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-53) [3c2dfb77] Attempting to update VMs/Templates Ovf. >2014-08-04 01:23:55,912 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-14) Principal admin::internal synchronized >2014-08-04 01:24:24,251 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-47) Attempting to update VMs/Templates Ovf. >2014-08-04 02:23:55,919 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-60) [1cb6c42c] Principal admin::internal synchronized >2014-08-04 02:24:24,263 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-15) [39225a1e] Attempting to update VMs/Templates Ovf. >2014-08-04 02:31:49,088 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-6) domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 in problem. vds: rack >2014-08-04 02:32:04,224 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-25) domain 44559ec0-6bef-491c-bbd6-e67278f37987:export in problem. vds: rack >2014-08-04 02:33:27,936 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-5) Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2014-08-04 02:33:27,941 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-5) [4f7e782e] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-08-04 02:33:27,945 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-5) [4f7e782e] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Active to Unknown, reason : null >2014-08-04 02:33:27,947 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-5) [4f7e782e] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Active to Unknown, reason : null >2014-08-04 02:33:27,960 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-5) [4f7e782e] Correlation ID: 4f7e782e, Call Stack: null, Custom Event ID: -1, Message: Invalid status on Data Center Default. Setting Data Center status to Non Responsive (On host rack, Error: Cannot access master storage domain). >2014-08-04 02:35:27,990 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-5) [4f7e782e] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2014-08-04 02:35:27,992 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-5) [4f7e782e] hostFromVds::selectedVds - rack, spmStatus returned null! >2014-08-04 02:35:27,993 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-5) [4f7e782e] IrsBroker::Failed::GetStoragePoolInfoVDS >2014-08-04 02:35:27,994 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-5) [4f7e782e] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2014-08-04 02:35:28,045 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-35) [c9f7d42] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage >2014-08-04 02:35:28,066 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-35) [c9f7d42] START, ResetIrsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, ignoreStopFailed = true), log id: 5dc8bb9b >2014-08-04 02:35:28,072 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-35) [c9f7d42] START, SpmStopVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9), log id: 3954a733 >2014-08-04 02:35:28,079 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-35) [c9f7d42] SpmStopVDSCommand::Stopping SPM on vds rack, pool id 00000002-0002-0002-0002-0000000001d9 >2014-08-04 02:37:27,961 WARN [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-35) [c9f7d42] could not stop spm of pool 00000002-0002-0002-0002-0000000001d9 on vds 17f7fe56-5f32-40ef-b098-611b84484e0b - reason: org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcRunTimeException: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException >2014-08-04 02:37:27,964 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-16) Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: VDSNetworkException: org.apache.commons.httpclient.NoHttpResponseException: The server 10.34.62.208 failed to respond >2014-08-04 02:37:27,970 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-16) hostFromVds::selectedVds - rack, spmStatus returned null! >2014-08-04 02:37:27,970 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ListVDSCommand] (DefaultQuartzScheduler_Worker-55) [7526d482] Command ListVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-08-04 02:37:27,973 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-55) [7526d482] Host rack is not responding. It will stay in Connecting state for a grace period of $80 seconds and after that an attempt to fence the host will be issued. >2014-08-04 02:37:27,967 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-35) [c9f7d42] FINISH, SpmStopVDSCommand, log id: 3954a733 >2014-08-04 02:37:27,975 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-55) [7526d482] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:406) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.fetchRunningVms(VdsUpdateRunTimeInfo.java:971) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVmStats(VdsUpdateRunTimeInfo.java:920) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:638) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:235) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) [:1.7.0_55] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_55] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_55] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_55] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_55] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_55] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_55] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_55] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy82.list(Unknown Source) > at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source) [:1.7.0_55] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_55] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_55] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_55] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_55] > >2014-08-04 02:37:27,987 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-35) [c9f7d42] FINISH, ResetIrsVDSCommand, log id: 5dc8bb9b >2014-08-04 02:37:28,014 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-55) [7526d482] Failed to refresh VDS , vds = 17f7fe56-5f32-40ef-b098-611b84484e0b : rack, VDS Network Error, continuing. >java.net.ConnectException: Connection refused >2014-08-04 02:37:28,019 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-35) [c9f7d42] Correlation ID: c9f7d42, Job ID: 8de3de07-7c75-47d5-8365-56a770348bad, Call Stack: null, Custom Event ID: -1, Message: Failed to Reconstruct Master Domain for Data Center Default. >2014-08-04 02:37:28,025 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-10) [4f7e782e] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue >2014-08-04 02:37:28,027 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-33) starting processDomainRecovery for domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 >2014-08-04 02:37:28,085 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-33) Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 was reported by all hosts in status UP as problematic. Not moving the domain to NonOperational because it is being reconstructed now. >2014-08-04 02:37:28,103 WARN [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-33) [64520b65] CanDoAction of action ReconstructMasterDomain failed. Reasons:VAR__ACTION__RECONSTRUCT_MASTER,VAR__TYPE__STORAGE__DOMAIN,ACTION_TYPE_FAILED_NO_VDS_IN_POOL >2014-08-04 02:37:28,109 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-33) [64520b65] Removing vds [] from the domain in maintenance cache >2014-08-04 02:37:28,110 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-33) [64520b65] Removing host(s) [] from hosts unseen domain report cache >2014-08-04 02:37:28,111 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-10) [4f7e782e] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue >2014-08-04 02:37:28,113 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-48) starting processDomainRecovery for domain 44559ec0-6bef-491c-bbd6-e67278f37987:export >2014-08-04 02:37:28,118 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-48) Domain 44559ec0-6bef-491c-bbd6-e67278f37987:export was reported by all hosts in status UP as problematic. Moving the domain to NonOperational. >2014-08-04 02:37:28,133 INFO [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-48) [3eb0d362] Lock Acquired to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE >, sharedLocks= ] >2014-08-04 02:37:28,139 INFO [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-48) [3eb0d362] Running command: DeactivateStorageDomainCommand internal: true. Entities affected : ID: 44559ec0-6bef-491c-bbd6-e67278f37987 Type: Storage >2014-08-04 02:37:28,145 INFO [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-48) [3eb0d362] Lock freed to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE >, sharedLocks= ] >2014-08-04 02:37:28,157 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-48) [3eb0d362] Correlation ID: 3eb0d362, Job ID: 5e8b9990-6fd8-4a56-98f0-b0349c058b39, Call Stack: null, Custom Event ID: -1, Message: Storage Domain export (Data Center Default) was deactivated by system because it's not visible by any of the hosts. >2014-08-04 02:37:28,163 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-48) [3eb0d362] Removing vds [] from the domain in maintenance cache >2014-08-04 02:37:28,164 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-48) [3eb0d362] Removing host(s) [] from hosts unseen domain report cache >2014-08-04 02:37:31,264 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-84) [59dec259] START, GetHardwareInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]), log id: 2809d9c7 >2014-08-04 02:37:31,319 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-84) [59dec259] FINISH, GetHardwareInfoVDSCommand, log id: 2809d9c7 >2014-08-04 02:37:31,345 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-84) [5165d85a] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS >2014-08-04 02:37:31,394 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-84) [2783c8a8] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-08-04 02:37:31,400 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-84) [760b6984] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-08-04 02:37:31,406 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-84) [760b6984] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, storageType = NFS, connectionList = [{ id: 05047c9a-152e-4ef2-b118-4a1753c1eae0, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35export, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 1f49a2d3 >2014-08-04 02:37:31,469 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-84) [760b6984] FINISH, ConnectStorageServerVDSCommand, return: {5083f370-af09-457b-9b73-3440e8b65a07=0, 05047c9a-152e-4ef2-b118-4a1753c1eae0=0}, log id: 1f49a2d3 >2014-08-04 02:37:31,470 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-84) [760b6984] Host rack storage connection was succeeded >2014-08-04 02:37:31,478 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-5) START, ConnectStoragePoolVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, masterVersion = 1), log id: 2bf13231 >2014-08-04 02:37:37,974 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-35) [5b4b3aec] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Inactive to Unknown, reason : no reporting hosts >2014-08-04 02:37:37,985 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-35) [5b4b3aec] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Inactive to Unknown, reason : no reporting hosts >2014-08-04 02:38:31,888 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-5) FINISH, ConnectStoragePoolVDSCommand, log id: 2bf13231 >2014-08-04 02:38:31,903 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-84) [760b6984] START, SetMOMPolicyParametersVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 7497473 >2014-08-04 02:38:32,049 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-84) [760b6984] FINISH, SetMOMPolicyParametersVDSCommand, log id: 7497473 >2014-08-04 02:38:32,056 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-84) [760b6984] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host rack was set to Up. >2014-08-04 02:38:32,062 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-84) [760b6984] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host rack. >2014-08-04 02:38:32,078 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-84) [53f4cdd5] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS >2014-08-04 02:38:38,141 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-89) [1b83a377] hostFromVds::selectedVds - rack, spmStatus Free, storage pool Default >2014-08-04 02:38:38,145 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-89) [1b83a377] starting spm on vds rack, storage pool Default, prevId -1, LVER -1 >2014-08-04 02:38:38,148 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-89) [1b83a377] START, SpmStartVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, prevId=-1, prevLVER=-1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 5fa85992 >2014-08-04 02:38:38,197 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-89) [1b83a377] spmStart polling started: taskId = cad5e411-9896-4abf-850d-6caea819d97e >2014-08-04 02:38:39,203 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-89) [1b83a377] spmStart polling ended: taskId = cad5e411-9896-4abf-850d-6caea819d97e task status = finished >2014-08-04 02:38:39,275 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-89) [1b83a377] spmStart polling ended, spm status: SPM >2014-08-04 02:38:39,277 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-89) [1b83a377] START, HSMClearTaskVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, taskId=cad5e411-9896-4abf-850d-6caea819d97e), log id: 73e43832 >2014-08-04 02:38:39,325 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-89) [1b83a377] FINISH, HSMClearTaskVDSCommand, log id: 73e43832 >2014-08-04 02:38:39,327 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-89) [1b83a377] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@7a94e565, log id: 5fa85992 >2014-08-04 02:38:39,382 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-89) [1b83a377] Initialize Irs proxy from vds: 10.34.62.208 >2014-08-04 02:38:39,393 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-89) [1b83a377] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208). >2014-08-04 02:38:39,394 WARN [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-8-thread-28) Executing a command: java.util.concurrent.FutureTask , but note that there are 1 tasks in the queue. >2014-08-04 02:38:39,406 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-28) [1b83a377] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 69415df4 >2014-08-04 02:38:39,470 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-28) [1b83a377] -- executeIrsBrokerCommand: Attempting on storage pool 00000002-0002-0002-0002-0000000001d9 >2014-08-04 02:38:39,471 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-28) [1b83a377] START, HSMGetAllTasksInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 518890f4 >2014-08-04 02:38:39,476 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-28) [1b83a377] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 518890f4 >2014-08-04 02:38:39,477 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-28) [1b83a377] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 69415df4 >2014-08-04 02:38:39,477 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (org.ovirt.thread.pool-8-thread-28) [1b83a377] Discovered no tasks on Storage Pool Default >2014-08-04 03:23:55,927 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-40) Principal admin::internal synchronized >2014-08-04 03:24:24,298 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-64) [686e64a1] Attempting to update VMs/Templates Ovf. >2014-08-04 03:35:35,001 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-26) [6bd9692] Start deleteAgedOutAuditLogs >2014-08-04 03:35:35,001 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-75) [6e437c9b] Start deleteAgedOutCommandEntities >2014-08-04 03:35:35,006 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-26) [6bd9692] Finished deleteAgedOutAuditLogs >2014-08-04 03:35:35,008 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-75) [6e437c9b] Finished deleteAgedOutCommandEntities >2014-08-04 04:23:55,934 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-90) Principal admin::internal synchronized >2014-08-04 04:24:24,332 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-20) [367a2664] Attempting to update VMs/Templates Ovf. >2014-08-04 05:23:55,940 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-53) [3c2dfb77] Principal admin::internal synchronized >2014-08-04 05:24:24,345 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-17) Attempting to update VMs/Templates Ovf. >2014-08-04 05:38:38,371 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-50) domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 in problem. vds: rack >2014-08-04 05:38:38,373 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-50) domain 44559ec0-6bef-491c-bbd6-e67278f37987:export in problem. vds: rack >2014-08-04 05:40:08,115 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-25) [50daf5eb] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2014-08-04 05:40:08,122 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-25) [32fc3079] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-08-04 05:40:08,124 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-25) [32fc3079] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Active to Unknown, reason : null >2014-08-04 05:40:08,126 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-25) [32fc3079] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Active to Unknown, reason : null >2014-08-04 05:40:08,136 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-25) [32fc3079] Correlation ID: 32fc3079, Call Stack: null, Custom Event ID: -1, Message: Invalid status on Data Center Default. Setting Data Center status to Non Responsive (On host rack, Error: Cannot access master storage domain). >2014-08-04 05:42:08,169 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-25) [32fc3079] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2014-08-04 05:42:08,173 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-25) [32fc3079] hostFromVds::selectedVds - rack, spmStatus returned null! >2014-08-04 05:42:08,174 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-25) [32fc3079] IrsBroker::Failed::GetStoragePoolInfoVDS >2014-08-04 05:42:08,175 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-25) [32fc3079] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2014-08-04 05:42:08,202 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-47) [2ae4fdd9] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage >2014-08-04 05:42:08,223 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-47) [2ae4fdd9] START, ResetIrsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, ignoreStopFailed = true), log id: 3cc190e4 >2014-08-04 05:42:08,227 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-47) [2ae4fdd9] START, SpmStopVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9), log id: 3f9e1a8c >2014-08-04 05:42:08,234 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-47) [2ae4fdd9] SpmStopVDSCommand::Stopping SPM on vds rack, pool id 00000002-0002-0002-0002-0000000001d9 >2014-08-04 05:44:08,142 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-73) [55405a19] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: VDSNetworkException: org.apache.commons.httpclient.NoHttpResponseException: The server 10.34.62.208 failed to respond >2014-08-04 05:44:08,144 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ListVDSCommand] (DefaultQuartzScheduler_Worker-2) Command ListVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-08-04 05:44:08,142 WARN [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-47) [2ae4fdd9] could not stop spm of pool 00000002-0002-0002-0002-0000000001d9 on vds 17f7fe56-5f32-40ef-b098-611b84484e0b - reason: org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcRunTimeException: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException >2014-08-04 05:44:08,150 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-47) [2ae4fdd9] FINISH, SpmStopVDSCommand, log id: 3f9e1a8c >2014-08-04 05:44:08,151 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-2) Host rack is not responding. It will stay in Connecting state for a grace period of $80 seconds and after that an attempt to fence the host will be issued. >2014-08-04 05:44:08,152 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-73) [55405a19] hostFromVds::selectedVds - rack, spmStatus returned null! >2014-08-04 05:44:08,152 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-2) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:406) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.fetchRunningVms(VdsUpdateRunTimeInfo.java:971) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVmStats(VdsUpdateRunTimeInfo.java:920) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:638) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:235) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) [:1.7.0_55] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_55] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_55] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_55] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_55] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_55] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_55] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_55] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy82.list(Unknown Source) > at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source) [:1.7.0_55] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_55] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_55] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_55] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_55] > >2014-08-04 05:44:08,160 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-47) [2ae4fdd9] FINISH, ResetIrsVDSCommand, log id: 3cc190e4 >2014-08-04 05:44:08,190 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-2) Failed to refresh VDS , vds = 17f7fe56-5f32-40ef-b098-611b84484e0b : rack, VDS Network Error, continuing. >java.net.ConnectException: Connection refused >2014-08-04 05:44:08,195 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-47) [2ae4fdd9] Correlation ID: 2ae4fdd9, Job ID: 55b2cd1f-f392-44ab-95c4-85d932d58c98, Call Stack: null, Custom Event ID: -1, Message: Failed to Reconstruct Master Domain for Data Center Default. >2014-08-04 05:44:08,203 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-8) [32fc3079] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue >2014-08-04 05:44:08,205 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-31) starting processDomainRecovery for domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 >2014-08-04 05:44:08,280 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-31) Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 was reported by all hosts in status UP as problematic. Not moving the domain to NonOperational because it is being reconstructed now. >2014-08-04 05:44:08,305 WARN [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-31) [37841764] CanDoAction of action ReconstructMasterDomain failed. Reasons:VAR__ACTION__RECONSTRUCT_MASTER,VAR__TYPE__STORAGE__DOMAIN,ACTION_TYPE_FAILED_NO_VDS_IN_POOL >2014-08-04 05:44:08,312 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-31) [37841764] Removing vds [] from the domain in maintenance cache >2014-08-04 05:44:08,313 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-31) [37841764] Removing host(s) [] from hosts unseen domain report cache >2014-08-04 05:44:08,314 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-8) [32fc3079] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue >2014-08-04 05:44:08,316 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-38) starting processDomainRecovery for domain 44559ec0-6bef-491c-bbd6-e67278f37987:export >2014-08-04 05:44:08,322 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-38) Domain 44559ec0-6bef-491c-bbd6-e67278f37987:export was reported by all hosts in status UP as problematic. Moving the domain to NonOperational. >2014-08-04 05:44:08,338 INFO [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-38) [283c9aab] Lock Acquired to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE >, sharedLocks= ] >2014-08-04 05:44:08,345 INFO [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-38) [283c9aab] Running command: DeactivateStorageDomainCommand internal: true. Entities affected : ID: 44559ec0-6bef-491c-bbd6-e67278f37987 Type: Storage >2014-08-04 05:44:08,351 INFO [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-38) [283c9aab] Lock freed to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE >, sharedLocks= ] >2014-08-04 05:44:08,361 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-38) [283c9aab] Correlation ID: 283c9aab, Job ID: c912c298-b426-4567-82d9-401dbefb5195, Call Stack: null, Custom Event ID: -1, Message: Storage Domain export (Data Center Default) was deactivated by system because it's not visible by any of the hosts. >2014-08-04 05:44:08,366 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-38) [283c9aab] Removing vds [] from the domain in maintenance cache >2014-08-04 05:44:08,367 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-38) [283c9aab] Removing host(s) [] from hosts unseen domain report cache >2014-08-04 05:44:11,450 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-39) [778da158] START, GetHardwareInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]), log id: 73e0104a >2014-08-04 05:44:11,504 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-39) [778da158] FINISH, GetHardwareInfoVDSCommand, log id: 73e0104a >2014-08-04 05:44:11,531 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-39) [4e3a2091] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS >2014-08-04 05:44:11,581 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-39) [5561bbe6] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-08-04 05:44:11,589 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-39) [5629f2c4] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-08-04 05:44:11,601 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-39) [5629f2c4] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, storageType = NFS, connectionList = [{ id: 05047c9a-152e-4ef2-b118-4a1753c1eae0, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35export, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 4e5be5aa >2014-08-04 05:44:11,663 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-39) [5629f2c4] FINISH, ConnectStorageServerVDSCommand, return: {5083f370-af09-457b-9b73-3440e8b65a07=0, 05047c9a-152e-4ef2-b118-4a1753c1eae0=0}, log id: 4e5be5aa >2014-08-04 05:44:11,664 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-39) [5629f2c4] Host rack storage connection was succeeded >2014-08-04 05:44:11,675 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-12) START, ConnectStoragePoolVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, masterVersion = 1), log id: a77f362 >2014-08-04 05:44:18,156 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-52) [8557062] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Inactive to Unknown, reason : no reporting hosts >2014-08-04 05:44:18,161 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-52) [8557062] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Inactive to Unknown, reason : no reporting hosts >2014-08-04 05:45:12,315 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-12) FINISH, ConnectStoragePoolVDSCommand, log id: a77f362 >2014-08-04 05:45:12,328 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-39) [5629f2c4] START, SetMOMPolicyParametersVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 1a63527c >2014-08-04 05:45:12,423 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-39) [5629f2c4] FINISH, SetMOMPolicyParametersVDSCommand, log id: 1a63527c >2014-08-04 05:45:12,430 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-39) [5629f2c4] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host rack was set to Up. >2014-08-04 05:45:12,436 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-39) [5629f2c4] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host rack. >2014-08-04 05:45:12,452 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-39) [2cd68be0] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS >2014-08-04 05:45:18,280 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-21) [81f884c] hostFromVds::selectedVds - rack, spmStatus Free, storage pool Default >2014-08-04 05:45:18,285 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-21) [81f884c] starting spm on vds rack, storage pool Default, prevId -1, LVER -1 >2014-08-04 05:45:18,288 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-21) [81f884c] START, SpmStartVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, prevId=-1, prevLVER=-1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 2dd3b127 >2014-08-04 05:45:18,337 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-21) [81f884c] spmStart polling started: taskId = 726e6e7a-639c-4042-bbea-8c35312f43ee >2014-08-04 05:45:19,344 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-21) [81f884c] spmStart polling ended: taskId = 726e6e7a-639c-4042-bbea-8c35312f43ee task status = finished >2014-08-04 05:45:19,400 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-21) [81f884c] spmStart polling ended, spm status: SPM >2014-08-04 05:45:19,403 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-21) [81f884c] START, HSMClearTaskVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, taskId=726e6e7a-639c-4042-bbea-8c35312f43ee), log id: 30ffa7ff >2014-08-04 05:45:19,449 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-21) [81f884c] FINISH, HSMClearTaskVDSCommand, log id: 30ffa7ff >2014-08-04 05:45:19,450 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-21) [81f884c] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@2995aa21, log id: 2dd3b127 >2014-08-04 05:45:19,455 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-21) [81f884c] Initialize Irs proxy from vds: 10.34.62.208 >2014-08-04 05:45:19,463 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-21) [81f884c] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208). >2014-08-04 05:45:19,465 WARN [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-8-thread-18) Executing a command: java.util.concurrent.FutureTask , but note that there are 1 tasks in the queue. >2014-08-04 05:45:19,484 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-18) [81f884c] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 32d8dd7b >2014-08-04 05:45:19,503 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-18) [81f884c] -- executeIrsBrokerCommand: Attempting on storage pool 00000002-0002-0002-0002-0000000001d9 >2014-08-04 05:45:19,505 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-18) [81f884c] START, HSMGetAllTasksInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 3bf0d2ea >2014-08-04 05:45:19,512 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-18) [81f884c] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 3bf0d2ea >2014-08-04 05:45:19,513 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-18) [81f884c] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 32d8dd7b >2014-08-04 05:45:19,514 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (org.ovirt.thread.pool-8-thread-18) [81f884c] Discovered no tasks on Storage Pool Default >2014-08-04 06:23:55,946 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-92) Principal admin::internal synchronized >2014-08-04 06:24:24,360 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-94) Attempting to update VMs/Templates Ovf. >2014-08-04 07:23:55,952 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-20) [367a2664] Principal admin::internal synchronized >2014-08-04 07:24:24,376 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-100) [3b9a9fa6] Attempting to update VMs/Templates Ovf. >2014-08-04 08:14:49,659 WARN [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-8-thread-50) Executing a command: java.util.concurrent.FutureTask , but note that there are 1 tasks in the queue. >2014-08-04 08:23:55,962 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-13) [519c108f] Principal admin::internal synchronized >2014-08-04 08:24:24,410 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-68) [4341002] Attempting to update VMs/Templates Ovf. >2014-08-04 08:45:13,493 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-29) domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 in problem. vds: rack >2014-08-04 08:45:13,495 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-29) domain 44559ec0-6bef-491c-bbd6-e67278f37987:export in problem. vds: rack >2014-08-04 08:46:40,490 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-95) [1abadf13] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2014-08-04 08:46:40,494 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-95) [6a286cf6] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-08-04 08:46:40,498 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-95) [6a286cf6] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Active to Unknown, reason : null >2014-08-04 08:46:40,500 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-95) [6a286cf6] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Active to Unknown, reason : null >2014-08-04 08:46:40,515 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-95) [6a286cf6] Correlation ID: 6a286cf6, Call Stack: null, Custom Event ID: -1, Message: Invalid status on Data Center Default. Setting Data Center status to Non Responsive (On host rack, Error: Cannot access master storage domain). >2014-08-04 08:48:40,542 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-95) [6a286cf6] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2014-08-04 08:48:40,544 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-95) [6a286cf6] hostFromVds::selectedVds - rack, spmStatus returned null! >2014-08-04 08:48:40,545 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-95) [6a286cf6] IrsBroker::Failed::GetStoragePoolInfoVDS >2014-08-04 08:48:40,547 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-95) [6a286cf6] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2014-08-04 08:48:40,582 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-41) [69d6e1dd] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage >2014-08-04 08:48:40,600 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-41) [69d6e1dd] START, ResetIrsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, ignoreStopFailed = true), log id: 71044258 >2014-08-04 08:48:40,604 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-41) [69d6e1dd] START, SpmStopVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9), log id: 49dccb43 >2014-08-04 08:48:40,609 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-41) [69d6e1dd] SpmStopVDSCommand::Stopping SPM on vds rack, pool id 00000002-0002-0002-0002-0000000001d9 >2014-08-04 08:50:00,005 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-87) Autorecovering 1 storage domains >2014-08-04 08:50:00,006 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-87) Autorecovering storage domains id: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 >2014-08-04 08:50:00,008 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-87) [129089c] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage >2014-08-04 08:50:00,010 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-87) [129089c] ConnectDomainToStorage. Before Connect all hosts to pool. Time:8/4/14 8:50 AM >2014-08-04 08:50:00,020 INFO [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-8-thread-10) [3eae806c] Running command: ConnectStorageToVdsCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2014-08-04 08:50:00,023 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-10) [3eae806c] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 787b77bb >2014-08-04 08:50:40,512 WARN [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-41) [69d6e1dd] could not stop spm of pool 00000002-0002-0002-0002-0000000001d9 on vds 17f7fe56-5f32-40ef-b098-611b84484e0b - reason: org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcRunTimeException: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException >2014-08-04 08:50:40,516 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-41) [69d6e1dd] FINISH, SpmStopVDSCommand, log id: 49dccb43 >2014-08-04 08:50:40,513 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ListVDSCommand] (DefaultQuartzScheduler_Worker-68) [4341002] Command ListVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-08-04 08:50:40,520 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-68) [4341002] Host rack is not responding. It will stay in Connecting state for a grace period of $80 seconds and after that an attempt to fence the host will be issued. >2014-08-04 08:50:40,514 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-37) Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: VDSNetworkException: org.apache.commons.httpclient.NoHttpResponseException: The server 10.34.62.208 failed to respond >2014-08-04 08:50:40,518 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-10) [3eae806c] Command ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-08-04 08:50:40,526 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-10) [3eae806c] FINISH, ConnectStorageServerVDSCommand, log id: 787b77bb >2014-08-04 08:50:40,527 ERROR [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-8-thread-10) [3eae806c] Command org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-08-04 08:50:40,523 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-37) hostFromVds::selectedVds - rack, spmStatus returned null! >2014-08-04 08:50:40,521 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-68) [4341002] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:406) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.fetchRunningVms(VdsUpdateRunTimeInfo.java:971) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVmStats(VdsUpdateRunTimeInfo.java:920) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:638) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:235) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) [:1.7.0_55] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_55] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_55] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_55] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_55] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_55] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_55] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_55] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy82.list(Unknown Source) > at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source) [:1.7.0_55] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_55] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_55] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_55] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_55] > >2014-08-04 08:50:40,528 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-41) [69d6e1dd] FINISH, ResetIrsVDSCommand, log id: 71044258 >2014-08-04 08:50:40,532 ERROR [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-8-thread-10) [3eae806c] Transaction rolled-back for command: org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand. >2014-08-04 08:50:40,555 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-87) [129089c] ConnectDomainToStorage. After Connect all hosts to pool. Time:8/4/14 8:50 AM >2014-08-04 08:50:40,562 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-68) [4341002] Failed to refresh VDS , vds = 17f7fe56-5f32-40ef-b098-611b84484e0b : rack, VDS Network Error, continuing. >java.net.ConnectException: Connection refused >2014-08-04 08:50:40,567 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-41) [69d6e1dd] Correlation ID: 69d6e1dd, Job ID: 9969f4b5-f80c-46ef-8f13-1be65ba93ce9, Call Stack: null, Custom Event ID: -1, Message: Failed to Reconstruct Master Domain for Data Center Default. >2014-08-04 08:50:40,575 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-5) [6a286cf6] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue >2014-08-04 08:50:40,577 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-19) starting processDomainRecovery for domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 >2014-08-04 08:50:40,652 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-19) Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 was reported by all hosts in status UP as problematic. Not moving the domain to NonOperational because it is being reconstructed now. >2014-08-04 08:50:40,679 WARN [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-19) [821b4e7] CanDoAction of action ReconstructMasterDomain failed. Reasons:VAR__ACTION__RECONSTRUCT_MASTER,VAR__TYPE__STORAGE__DOMAIN,ACTION_TYPE_FAILED_NO_VDS_IN_POOL >2014-08-04 08:50:40,688 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-19) [821b4e7] Removing vds [] from the domain in maintenance cache >2014-08-04 08:50:40,689 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-19) [821b4e7] Removing host(s) [] from hosts unseen domain report cache >2014-08-04 08:50:40,690 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-5) [6a286cf6] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue >2014-08-04 08:50:40,692 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-17) starting processDomainRecovery for domain 44559ec0-6bef-491c-bbd6-e67278f37987:export >2014-08-04 08:50:40,698 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-17) Domain 44559ec0-6bef-491c-bbd6-e67278f37987:export was reported by all hosts in status UP as problematic. Moving the domain to NonOperational. >2014-08-04 08:50:40,713 INFO [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-17) [60237d1f] Lock Acquired to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE >, sharedLocks= ] >2014-08-04 08:50:40,720 INFO [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-17) [60237d1f] Running command: DeactivateStorageDomainCommand internal: true. Entities affected : ID: 44559ec0-6bef-491c-bbd6-e67278f37987 Type: Storage >2014-08-04 08:50:40,727 INFO [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-17) [60237d1f] Lock freed to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE >, sharedLocks= ] >2014-08-04 08:50:40,737 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-17) [60237d1f] Correlation ID: 60237d1f, Job ID: bd045107-439b-4448-9f13-33fac5371203, Call Stack: null, Custom Event ID: -1, Message: Storage Domain export (Data Center Default) was deactivated by system because it's not visible by any of the hosts. >2014-08-04 08:50:40,742 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-17) [60237d1f] Removing vds [] from the domain in maintenance cache >2014-08-04 08:50:40,743 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-17) [60237d1f] Removing host(s) [] from hosts unseen domain report cache >2014-08-04 08:50:43,657 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-25) [32fc3079] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@793c69e1 >2014-08-04 08:50:43,659 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-25) [32fc3079] HostName = rack >2014-08-04 08:50:43,660 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-25) [32fc3079] Command GetCapabilitiesVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]) execution failed. Exception: VDSRecoveringException: Recovering from crash or Initializing >2014-08-04 08:50:43,674 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-25) [32fc3079] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSRecoveringException: Recovering from crash or Initializing > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:16) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:566) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:235) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Host rack is initializing. Message: Recovering from crash or Initializing >2014-08-04 08:50:43,680 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-25) [32fc3079] Failed to refresh VDS , vds = 17f7fe56-5f32-40ef-b098-611b84484e0b : rack, error = Recovering from crash or Initializing, continuing. >2014-08-04 08:50:46,908 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-85) [3e29dd44] START, GetHardwareInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]), log id: 45e49805 >2014-08-04 08:50:46,961 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-85) [3e29dd44] FINISH, GetHardwareInfoVDSCommand, log id: 45e49805 >2014-08-04 08:50:46,984 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-85) [156f0be3] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS >2014-08-04 08:50:47,033 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-85) [156f0be3] Cancelling the recovery from crash timer for VDS 17f7fe56-5f32-40ef-b098-611b84484e0b because vds started initializing >2014-08-04 08:50:47,035 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-85) [59ba3d1e] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-08-04 08:50:47,043 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-85) [149b0156] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-08-04 08:50:47,055 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-85) [149b0156] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, storageType = NFS, connectionList = [{ id: 05047c9a-152e-4ef2-b118-4a1753c1eae0, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35export, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 1cbfa1dc >2014-08-04 08:50:47,119 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-85) [149b0156] FINISH, ConnectStorageServerVDSCommand, return: {5083f370-af09-457b-9b73-3440e8b65a07=0, 05047c9a-152e-4ef2-b118-4a1753c1eae0=0}, log id: 1cbfa1dc >2014-08-04 08:50:47,120 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-85) [149b0156] Host rack storage connection was succeeded >2014-08-04 08:50:47,129 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-46) START, ConnectStoragePoolVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, masterVersion = 1), log id: 65528e10 >2014-08-04 08:50:50,534 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-92) Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Inactive to Unknown, reason : no reporting hosts >2014-08-04 08:50:50,537 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-92) Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Inactive to Unknown, reason : no reporting hosts >2014-08-04 08:51:47,520 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-46) FINISH, ConnectStoragePoolVDSCommand, log id: 65528e10 >2014-08-04 08:51:47,538 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-85) [149b0156] START, SetMOMPolicyParametersVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 79ef92b2 >2014-08-04 08:51:47,634 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-85) [149b0156] FINISH, SetMOMPolicyParametersVDSCommand, log id: 79ef92b2 >2014-08-04 08:51:47,645 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-85) [149b0156] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host rack was set to Up. >2014-08-04 08:51:47,653 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-85) [149b0156] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host rack. >2014-08-04 08:51:47,671 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-85) [9016fac] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS >2014-08-04 08:51:50,668 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-6) [7f1e369e] hostFromVds::selectedVds - rack, spmStatus Free, storage pool Default >2014-08-04 08:51:50,675 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-6) [7f1e369e] starting spm on vds rack, storage pool Default, prevId -1, LVER -1 >2014-08-04 08:51:50,678 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-6) [7f1e369e] START, SpmStartVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, prevId=-1, prevLVER=-1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 66cfa6c3 >2014-08-04 08:51:50,729 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-6) [7f1e369e] spmStart polling started: taskId = 456c1d7c-a0e4-4842-8e00-90faa50e28ef >2014-08-04 08:51:51,735 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-6) [7f1e369e] spmStart polling ended: taskId = 456c1d7c-a0e4-4842-8e00-90faa50e28ef task status = finished >2014-08-04 08:51:51,790 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-6) [7f1e369e] spmStart polling ended, spm status: SPM >2014-08-04 08:51:51,793 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-6) [7f1e369e] START, HSMClearTaskVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, taskId=456c1d7c-a0e4-4842-8e00-90faa50e28ef), log id: 64e69ea5 >2014-08-04 08:51:51,839 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-6) [7f1e369e] FINISH, HSMClearTaskVDSCommand, log id: 64e69ea5 >2014-08-04 08:51:51,840 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-6) [7f1e369e] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@1835faa8, log id: 66cfa6c3 >2014-08-04 08:51:51,845 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-6) [7f1e369e] Initialize Irs proxy from vds: 10.34.62.208 >2014-08-04 08:51:51,854 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-6) [7f1e369e] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208). >2014-08-04 08:51:51,872 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-43) [7f1e369e] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 49bb32b6 >2014-08-04 08:51:51,904 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-43) [7f1e369e] -- executeIrsBrokerCommand: Attempting on storage pool 00000002-0002-0002-0002-0000000001d9 >2014-08-04 08:51:51,906 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-43) [7f1e369e] START, HSMGetAllTasksInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 68c432cc >2014-08-04 08:51:51,911 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-43) [7f1e369e] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 68c432cc >2014-08-04 08:51:51,912 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-43) [7f1e369e] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 49bb32b6 >2014-08-04 08:51:51,913 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (org.ovirt.thread.pool-8-thread-43) [7f1e369e] Discovered no tasks on Storage Pool Default >2014-08-04 09:23:55,969 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-47) Principal admin::internal synchronized >2014-08-04 09:24:24,422 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-29) [327b509d] Attempting to update VMs/Templates Ovf. >2014-08-04 10:23:55,973 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-14) Principal admin::internal synchronized >2014-08-04 10:24:24,438 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-61) [58bfe349] Attempting to update VMs/Templates Ovf. >2014-08-04 11:23:55,979 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-47) Principal admin::internal synchronized >2014-08-04 11:24:24,454 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-100) [3b9a9fa6] Attempting to update VMs/Templates Ovf. >2014-08-04 11:51:33,310 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-45) domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 in problem. vds: rack >2014-08-04 11:51:48,437 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-38) domain 44559ec0-6bef-491c-bbd6-e67278f37987:export in problem. vds: rack >2014-08-04 11:53:10,164 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-80) [78845858] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2014-08-04 11:53:10,168 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-80) [386bc8c5] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-08-04 11:53:10,172 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-80) [386bc8c5] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Active to Unknown, reason : null >2014-08-04 11:53:10,174 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-80) [386bc8c5] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Active to Unknown, reason : null >2014-08-04 11:53:10,191 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-80) [386bc8c5] Correlation ID: 386bc8c5, Call Stack: null, Custom Event ID: -1, Message: Invalid status on Data Center Default. Setting Data Center status to Non Responsive (On host rack, Error: Cannot access master storage domain). >2014-08-04 11:55:10,226 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-80) [386bc8c5] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2014-08-04 11:55:10,228 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-80) [386bc8c5] hostFromVds::selectedVds - rack, spmStatus returned null! >2014-08-04 11:55:10,228 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-80) [386bc8c5] IrsBroker::Failed::GetStoragePoolInfoVDS >2014-08-04 11:55:10,229 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-80) [386bc8c5] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2014-08-04 11:55:10,256 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-36) [343a7b93] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage >2014-08-04 11:55:10,268 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-36) [343a7b93] START, ResetIrsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, ignoreStopFailed = true), log id: 21954c7d >2014-08-04 11:55:10,270 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-36) [343a7b93] START, SpmStopVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9), log id: 77dc8f79 >2014-08-04 11:55:10,280 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-36) [343a7b93] SpmStopVDSCommand::Stopping SPM on vds rack, pool id 00000002-0002-0002-0002-0000000001d9 >2014-08-04 11:57:10,247 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-93) [71be4699] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2014-08-04 11:57:10,249 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-93) [71be4699] hostFromVds::selectedVds - rack, spmStatus returned null! >2014-08-04 11:57:10,254 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-93) [71be4699] IrsBroker::Failed::GetStoragePoolInfoVDS >2014-08-04 11:57:10,258 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-93) [71be4699] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2014-08-04 11:58:10,281 WARN [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-36) [343a7b93] could not stop spm of pool 00000002-0002-0002-0002-0000000001d9 on vds 17f7fe56-5f32-40ef-b098-611b84484e0b - reason: org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcRunTimeException: java.util.concurrent.TimeoutException >2014-08-04 11:58:10,283 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-36) [343a7b93] FINISH, SpmStopVDSCommand, log id: 77dc8f79 >2014-08-04 11:58:10,289 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-36) [343a7b93] FINISH, ResetIrsVDSCommand, log id: 21954c7d >2014-08-04 11:58:10,298 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-36) [343a7b93] Correlation ID: 343a7b93, Job ID: e798785d-fe15-45d8-91b4-164d267380eb, Call Stack: null, Custom Event ID: -1, Message: Failed to Reconstruct Master Domain for Data Center Default. >2014-08-04 11:58:10,304 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-43) [386bc8c5] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue >2014-08-04 11:58:10,306 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-38) starting processDomainRecovery for domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 >2014-08-04 11:58:10,358 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-38) Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 was reported by all hosts in status UP as problematic. Not moving the domain to NonOperational because it is being reconstructed now. >2014-08-04 11:58:10,383 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-38) [6872f8b3] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage >2014-08-04 11:58:10,428 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-38) [6872f8b3] Removing vds [] from the domain in maintenance cache >2014-08-04 11:58:10,429 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-38) [6872f8b3] Removing host(s) [] from hosts unseen domain report cache >2014-08-04 11:58:10,430 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-43) [386bc8c5] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue >2014-08-04 11:58:10,432 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-37) starting processDomainRecovery for domain 44559ec0-6bef-491c-bbd6-e67278f37987:export >2014-08-04 11:58:10,436 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-37) Domain 44559ec0-6bef-491c-bbd6-e67278f37987:export was reported by all hosts in status UP as problematic. Moving the domain to NonOperational. >2014-08-04 11:58:10,451 INFO [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-37) [37fc09d8] Lock Acquired to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE >, sharedLocks= ] >2014-08-04 11:58:10,457 INFO [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-37) [37fc09d8] Running command: DeactivateStorageDomainCommand internal: true. Entities affected : ID: 44559ec0-6bef-491c-bbd6-e67278f37987 Type: Storage >2014-08-04 11:58:10,464 INFO [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-37) [37fc09d8] Lock freed to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE >, sharedLocks= ] >2014-08-04 11:58:10,477 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-37) [37fc09d8] Correlation ID: 37fc09d8, Job ID: f8c0d157-d357-498f-baf1-3fec31e86ac2, Call Stack: null, Custom Event ID: -1, Message: Storage Domain export (Data Center Default) was deactivated by system because it's not visible by any of the hosts. >2014-08-04 11:58:10,486 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-37) [37fc09d8] Removing vds [] from the domain in maintenance cache >2014-08-04 11:58:10,487 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-37) [37fc09d8] Removing host(s) [] from hosts unseen domain report cache >2014-08-04 11:59:10,222 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-50) [16080eb6] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: VDSNetworkException: org.apache.commons.httpclient.NoHttpResponseException: The server 10.34.62.208 failed to respond >2014-08-04 11:59:10,225 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-50) [16080eb6] hostFromVds::selectedVds - rack, spmStatus returned null! >2014-08-04 11:59:10,995 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ListVDSCommand] (DefaultQuartzScheduler_Worker-11) [aeb0141] Command ListVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]) execution failed. Exception: VDSRecoveringException: Recovering from crash or Initializing >2014-08-04 11:59:10,996 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-11) [aeb0141] Failed vds listing, vds = 17f7fe56-5f32-40ef-b098-611b84484e0b : rack, error = org.ovirt.engine.core.vdsbroker.vdsbroker.VDSRecoveringException: Recovering from crash or Initializing >2014-08-04 11:59:11,011 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-11) [aeb0141] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSRecoveringException: Recovering from crash or Initializing > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.vdsbroker.ListVDSCommand.executeVdsBrokerCommand(ListVDSCommand.java:24) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:406) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.fetchRunningVms(VdsUpdateRunTimeInfo.java:971) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVmStats(VdsUpdateRunTimeInfo.java:920) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:638) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:235) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Host rack is initializing. Message: Recovering from crash or Initializing >2014-08-04 11:59:11,019 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-11) [aeb0141] Failed to refresh VDS , vds = 17f7fe56-5f32-40ef-b098-611b84484e0b : rack, error = Recovering from crash or Initializing, continuing. >2014-08-04 11:59:14,256 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-37) START, GetHardwareInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]), log id: 21282722 >2014-08-04 11:59:14,312 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-37) FINISH, GetHardwareInfoVDSCommand, log id: 21282722 >2014-08-04 11:59:14,338 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-37) [6fb36f9] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS >2014-08-04 11:59:14,352 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-37) [6fb36f9] Cancelling the recovery from crash timer for VDS 17f7fe56-5f32-40ef-b098-611b84484e0b because vds started initializing >2014-08-04 11:59:14,356 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-37) [25c63078] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-08-04 11:59:14,364 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-37) [2c6893c] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-08-04 11:59:14,370 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-37) [2c6893c] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, storageType = NFS, connectionList = [{ id: 05047c9a-152e-4ef2-b118-4a1753c1eae0, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35export, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 63b33708 >2014-08-04 11:59:14,429 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-37) [2c6893c] FINISH, ConnectStorageServerVDSCommand, return: {5083f370-af09-457b-9b73-3440e8b65a07=0, 05047c9a-152e-4ef2-b118-4a1753c1eae0=0}, log id: 63b33708 >2014-08-04 11:59:14,430 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-37) [2c6893c] Host rack storage connection was succeeded >2014-08-04 11:59:14,437 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-41) START, ConnectStoragePoolVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, masterVersion = 1), log id: 5bea21b >2014-08-04 11:59:20,229 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-64) [686e64a1] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Inactive to Unknown, reason : no reporting hosts >2014-08-04 11:59:20,233 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-64) [686e64a1] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Inactive to Unknown, reason : no reporting hosts >2014-08-04 12:00:14,857 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-41) FINISH, ConnectStoragePoolVDSCommand, log id: 5bea21b >2014-08-04 12:00:14,871 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-37) [2c6893c] START, SetMOMPolicyParametersVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 1847737c >2014-08-04 12:00:14,963 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-37) [2c6893c] FINISH, SetMOMPolicyParametersVDSCommand, log id: 1847737c >2014-08-04 12:00:14,970 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-37) [2c6893c] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host rack was set to Up. >2014-08-04 12:00:14,976 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-37) [2c6893c] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host rack. >2014-08-04 12:00:14,999 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-37) [68f898a6] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS >2014-08-04 12:00:20,342 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-34) [36ef1be6] hostFromVds::selectedVds - rack, spmStatus Free, storage pool Default >2014-08-04 12:00:20,348 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-34) [36ef1be6] starting spm on vds rack, storage pool Default, prevId -1, LVER -1 >2014-08-04 12:00:20,351 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-34) [36ef1be6] START, SpmStartVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, prevId=-1, prevLVER=-1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 18f55877 >2014-08-04 12:00:20,401 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-34) [36ef1be6] spmStart polling started: taskId = 22eed789-2516-4fcd-97a0-0a74cd979b9c >2014-08-04 12:00:21,408 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-34) [36ef1be6] spmStart polling ended: taskId = 22eed789-2516-4fcd-97a0-0a74cd979b9c task status = finished >2014-08-04 12:00:21,465 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-34) [36ef1be6] spmStart polling ended, spm status: SPM >2014-08-04 12:00:21,467 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-34) [36ef1be6] START, HSMClearTaskVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, taskId=22eed789-2516-4fcd-97a0-0a74cd979b9c), log id: 15425cd5 >2014-08-04 12:00:21,512 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-34) [36ef1be6] FINISH, HSMClearTaskVDSCommand, log id: 15425cd5 >2014-08-04 12:00:21,513 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-34) [36ef1be6] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@7965adf, log id: 18f55877 >2014-08-04 12:00:21,518 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-34) [36ef1be6] Initialize Irs proxy from vds: 10.34.62.208 >2014-08-04 12:00:21,525 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-34) [36ef1be6] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208). >2014-08-04 12:00:21,526 WARN [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-8-thread-21) Executing a command: java.util.concurrent.FutureTask , but note that there are 1 tasks in the queue. >2014-08-04 12:00:21,541 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-21) [36ef1be6] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 78f38c1a >2014-08-04 12:00:21,566 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-21) [36ef1be6] -- executeIrsBrokerCommand: Attempting on storage pool 00000002-0002-0002-0002-0000000001d9 >2014-08-04 12:00:21,568 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-21) [36ef1be6] START, HSMGetAllTasksInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 315f0ec3 >2014-08-04 12:00:21,573 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-21) [36ef1be6] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 315f0ec3 >2014-08-04 12:00:21,574 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-21) [36ef1be6] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 78f38c1a >2014-08-04 12:00:21,575 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (org.ovirt.thread.pool-8-thread-21) [36ef1be6] Discovered no tasks on Storage Pool Default >2014-08-04 12:23:55,986 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-82) [3d60c3e4] Principal admin::internal synchronized >2014-08-04 12:24:24,469 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-48) [14cac44e] Attempting to update VMs/Templates Ovf. >2014-08-04 13:23:55,993 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-47) Principal admin::internal synchronized >2014-08-04 13:24:24,509 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-32) Attempting to update VMs/Templates Ovf. >2014-08-04 14:23:55,999 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-27) Principal admin::internal synchronized >2014-08-04 14:24:24,523 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-62) [2cdbf22e] Attempting to update VMs/Templates Ovf. >2014-08-04 14:59:45,039 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp--127.0.0.1-8702-1) Running command: LoginUserCommand internal: false. >2014-08-04 14:59:45,047 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-1) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-08-04 14:59:45,058 INFO [org.ovirt.engine.core.bll.LogoutBySessionCommand] (ajp--127.0.0.1-8702-1) [663a0e55] Running command: LogoutBySessionCommand internal: false. >2014-08-04 14:59:45,062 INFO [org.ovirt.engine.core.bll.LogoutUserCommand] (ajp--127.0.0.1-8702-1) [3f6766bd] Running command: LogoutUserCommand internal: false. >2014-08-04 14:59:45,067 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-1) [3f6766bd] Correlation ID: 3f6766bd, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-08-04 15:00:11,638 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-33) domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 in problem. vds: rack >2014-08-04 15:00:11,640 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-33) domain 44559ec0-6bef-491c-bbd6-e67278f37987:export in problem. vds: rack >2014-08-04 15:01:02,323 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp--127.0.0.1-8702-11) Running command: LoginUserCommand internal: false. >2014-08-04 15:01:02,328 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-11) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-08-04 15:01:02,337 INFO [org.ovirt.engine.core.bll.LogoutBySessionCommand] (ajp--127.0.0.1-8702-11) [1299c980] Running command: LogoutBySessionCommand internal: false. >2014-08-04 15:01:02,342 INFO [org.ovirt.engine.core.bll.LogoutUserCommand] (ajp--127.0.0.1-8702-11) [6fe70835] Running command: LogoutUserCommand internal: false. >2014-08-04 15:01:02,348 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-11) [6fe70835] Correlation ID: 6fe70835, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-08-04 15:01:38,514 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-78) Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2014-08-04 15:01:38,518 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-78) [412fb852] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-08-04 15:01:38,521 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-78) [412fb852] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Active to Unknown, reason : null >2014-08-04 15:01:38,523 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-78) [412fb852] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Active to Unknown, reason : null >2014-08-04 15:01:38,535 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-78) [412fb852] Correlation ID: 412fb852, Call Stack: null, Custom Event ID: -1, Message: Invalid status on Data Center Default. Setting Data Center status to Non Responsive (On host rack, Error: Cannot access master storage domain). >2014-08-04 15:03:15,499 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp--127.0.0.1-8702-2) Running command: LoginUserCommand internal: false. >2014-08-04 15:03:15,504 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-2) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-08-04 15:03:15,510 INFO [org.ovirt.engine.core.bll.LogoutBySessionCommand] (ajp--127.0.0.1-8702-2) [1835f83c] Running command: LogoutBySessionCommand internal: false. >2014-08-04 15:03:15,513 INFO [org.ovirt.engine.core.bll.LogoutUserCommand] (ajp--127.0.0.1-8702-2) [1f8a4f63] Running command: LogoutUserCommand internal: false. >2014-08-04 15:03:15,517 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-2) [1f8a4f63] Correlation ID: 1f8a4f63, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-08-04 15:03:38,572 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-78) [412fb852] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2014-08-04 15:03:38,580 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-78) [412fb852] hostFromVds::selectedVds - rack, spmStatus returned null! >2014-08-04 15:03:38,584 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-78) [412fb852] IrsBroker::Failed::GetStoragePoolInfoVDS >2014-08-04 15:03:38,587 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-78) [412fb852] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2014-08-04 15:03:38,625 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-9) [473fd2de] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage >2014-08-04 15:03:38,663 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-9) [473fd2de] START, ResetIrsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, ignoreStopFailed = true), log id: 668fe626 >2014-08-04 15:03:38,670 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-9) [473fd2de] START, SpmStopVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9), log id: 164dcfc0 >2014-08-04 15:03:38,688 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-9) [473fd2de] SpmStopVDSCommand::Stopping SPM on vds rack, pool id 00000002-0002-0002-0002-0000000001d9 >2014-08-04 15:04:55,393 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp--127.0.0.1-8702-3) Running command: LoginUserCommand internal: false. >2014-08-04 15:04:55,397 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-3) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-08-04 15:04:55,404 INFO [org.ovirt.engine.core.bll.LogoutBySessionCommand] (ajp--127.0.0.1-8702-3) [22c856e0] Running command: LogoutBySessionCommand internal: false. >2014-08-04 15:04:55,418 INFO [org.ovirt.engine.core.bll.LogoutUserCommand] (ajp--127.0.0.1-8702-3) [7f1f6a27] Running command: LogoutUserCommand internal: false. >2014-08-04 15:04:55,423 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-3) [7f1f6a27] Correlation ID: 7f1f6a27, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-08-04 15:05:00,006 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-33) [6bf204c4] Autorecovering 1 storage domains >2014-08-04 15:05:00,007 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-33) [6bf204c4] Autorecovering storage domains id: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 >2014-08-04 15:05:00,009 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-33) [1f9421b1] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage >2014-08-04 15:05:00,011 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-33) [1f9421b1] ConnectDomainToStorage. Before Connect all hosts to pool. Time:8/4/14 3:05 PM >2014-08-04 15:05:00,021 INFO [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-8-thread-42) [50eb80e6] Running command: ConnectStorageToVdsCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2014-08-04 15:05:00,025 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-42) [50eb80e6] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 5fde6970 >2014-08-04 15:05:38,538 WARN [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-9) [473fd2de] could not stop spm of pool 00000002-0002-0002-0002-0000000001d9 on vds 17f7fe56-5f32-40ef-b098-611b84484e0b - reason: org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcRunTimeException: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException >2014-08-04 15:05:38,541 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-9) [473fd2de] FINISH, SpmStopVDSCommand, log id: 164dcfc0 >2014-08-04 15:05:38,545 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-27) Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: VDSNetworkException: org.apache.commons.httpclient.NoHttpResponseException: The server 10.34.62.208 failed to respond >2014-08-04 15:05:38,548 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-27) hostFromVds::selectedVds - rack, spmStatus returned null! >2014-08-04 15:05:38,550 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-42) [50eb80e6] Command ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-08-04 15:05:38,550 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ListVDSCommand] (DefaultQuartzScheduler_Worker-7) [42c2440e] Command ListVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-08-04 15:05:38,554 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-42) [50eb80e6] FINISH, ConnectStorageServerVDSCommand, log id: 5fde6970 >2014-08-04 15:05:38,557 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-9) [473fd2de] FINISH, ResetIrsVDSCommand, log id: 668fe626 >2014-08-04 15:05:38,556 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-7) [42c2440e] Host rack is not responding. It will stay in Connecting state for a grace period of $80 seconds and after that an attempt to fence the host will be issued. >2014-08-04 15:05:38,557 ERROR [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-8-thread-42) [50eb80e6] Command org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-08-04 15:05:38,560 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-7) [42c2440e] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:406) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.fetchRunningVms(VdsUpdateRunTimeInfo.java:971) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVmStats(VdsUpdateRunTimeInfo.java:920) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:638) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:235) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) [:1.7.0_55] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_55] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_55] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_55] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_55] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_55] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_55] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_55] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy82.list(Unknown Source) > at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source) [:1.7.0_55] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_55] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_55] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_55] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_55] > >2014-08-04 15:05:38,572 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-9) [473fd2de] Correlation ID: 473fd2de, Job ID: 43068271-5fdf-448f-bb9f-441b15d0aaab, Call Stack: null, Custom Event ID: -1, Message: Failed to Reconstruct Master Domain for Data Center Default. >2014-08-04 15:05:38,564 ERROR [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-8-thread-42) [50eb80e6] Transaction rolled-back for command: org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand. >2014-08-04 15:05:38,593 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-33) [1f9421b1] ConnectDomainToStorage. After Connect all hosts to pool. Time:8/4/14 3:05 PM >2014-08-04 15:05:38,596 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-7) [42c2440e] Failed to refresh VDS , vds = 17f7fe56-5f32-40ef-b098-611b84484e0b : rack, VDS Network Error, continuing. >java.net.ConnectException: Connection refused >2014-08-04 15:05:38,599 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-48) [412fb852] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue >2014-08-04 15:05:38,600 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-21) starting processDomainRecovery for domain 44559ec0-6bef-491c-bbd6-e67278f37987:export >2014-08-04 15:05:38,665 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-21) Domain 44559ec0-6bef-491c-bbd6-e67278f37987:export was reported by all hosts in status UP as problematic. Moving the domain to NonOperational. >2014-08-04 15:05:38,680 INFO [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-21) [3b604dc6] Lock Acquired to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE >, sharedLocks= ] >2014-08-04 15:05:38,688 INFO [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-21) [3b604dc6] Running command: DeactivateStorageDomainCommand internal: true. Entities affected : ID: 44559ec0-6bef-491c-bbd6-e67278f37987 Type: Storage >2014-08-04 15:05:38,693 INFO [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-21) [3b604dc6] Lock freed to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE >, sharedLocks= ] >2014-08-04 15:05:38,702 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-21) [3b604dc6] Correlation ID: 3b604dc6, Job ID: 39312fcf-aa1f-4856-af2a-ccaf564ecc60, Call Stack: null, Custom Event ID: -1, Message: Storage Domain export (Data Center Default) was deactivated by system because it's not visible by any of the hosts. >2014-08-04 15:05:38,708 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-21) [3b604dc6] Removing vds [] from the domain in maintenance cache >2014-08-04 15:05:38,708 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-21) [3b604dc6] Removing host(s) [] from hosts unseen domain report cache >2014-08-04 15:05:38,710 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-20) starting processDomainRecovery for domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 >2014-08-04 15:05:38,715 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-20) Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 was reported by all hosts in status UP as problematic. Not moving the domain to NonOperational because it is being reconstructed now. >2014-08-04 15:05:38,731 WARN [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-20) [397d3cb3] CanDoAction of action ReconstructMasterDomain failed. Reasons:VAR__ACTION__RECONSTRUCT_MASTER,VAR__TYPE__STORAGE__DOMAIN,ACTION_TYPE_FAILED_NO_VDS_IN_POOL >2014-08-04 15:05:38,737 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-20) [397d3cb3] Removing vds [] from the domain in maintenance cache >2014-08-04 15:05:38,738 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-20) [397d3cb3] Removing host(s) [] from hosts unseen domain report cache >2014-08-04 15:05:38,739 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-48) [412fb852] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue >2014-08-04 15:05:41,872 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-74) START, GetHardwareInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]), log id: 155034f3 >2014-08-04 15:05:41,929 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-74) FINISH, GetHardwareInfoVDSCommand, log id: 155034f3 >2014-08-04 15:05:41,971 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-74) [56bacfd3] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS >2014-08-04 15:05:41,981 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-74) [458b4be2] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-08-04 15:05:41,989 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-74) [32674fb8] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-08-04 15:05:42,000 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-74) [32674fb8] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, storageType = NFS, connectionList = [{ id: 05047c9a-152e-4ef2-b118-4a1753c1eae0, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35export, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 40e5cf30 >2014-08-04 15:05:42,063 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-74) [32674fb8] FINISH, ConnectStorageServerVDSCommand, return: {5083f370-af09-457b-9b73-3440e8b65a07=0, 05047c9a-152e-4ef2-b118-4a1753c1eae0=0}, log id: 40e5cf30 >2014-08-04 15:05:42,064 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-74) [32674fb8] Host rack storage connection was succeeded >2014-08-04 15:05:42,073 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-31) START, ConnectStoragePoolVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, masterVersion = 1), log id: 7b101cd7 >2014-08-04 15:05:48,555 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-29) [327b509d] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Inactive to Unknown, reason : no reporting hosts >2014-08-04 15:05:48,558 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-29) [327b509d] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Inactive to Unknown, reason : no reporting hosts >2014-08-04 15:06:42,468 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-31) FINISH, ConnectStoragePoolVDSCommand, log id: 7b101cd7 >2014-08-04 15:06:42,480 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-74) [32674fb8] START, SetMOMPolicyParametersVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 39925c2c >2014-08-04 15:06:42,573 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-74) [32674fb8] FINISH, SetMOMPolicyParametersVDSCommand, log id: 39925c2c >2014-08-04 15:06:42,580 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-74) [32674fb8] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host rack was set to Up. >2014-08-04 15:06:42,587 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-74) [32674fb8] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host rack. >2014-08-04 15:06:42,603 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-74) [5972dc2] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS >2014-08-04 15:06:48,716 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-84) [53f4cdd5] hostFromVds::selectedVds - rack, spmStatus Free, storage pool Default >2014-08-04 15:06:48,723 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-84) [53f4cdd5] starting spm on vds rack, storage pool Default, prevId -1, LVER -1 >2014-08-04 15:06:48,725 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-84) [53f4cdd5] START, SpmStartVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, prevId=-1, prevLVER=-1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 46b44774 >2014-08-04 15:06:48,774 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-84) [53f4cdd5] spmStart polling started: taskId = a43d14d6-dac0-44ef-a3a9-588e590388a4 >2014-08-04 15:06:49,781 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-84) [53f4cdd5] spmStart polling ended: taskId = a43d14d6-dac0-44ef-a3a9-588e590388a4 task status = finished >2014-08-04 15:06:49,838 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-84) [53f4cdd5] spmStart polling ended, spm status: SPM >2014-08-04 15:06:49,840 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-84) [53f4cdd5] START, HSMClearTaskVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, taskId=a43d14d6-dac0-44ef-a3a9-588e590388a4), log id: 4435c065 >2014-08-04 15:06:49,885 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-84) [53f4cdd5] FINISH, HSMClearTaskVDSCommand, log id: 4435c065 >2014-08-04 15:06:49,886 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-84) [53f4cdd5] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@3e88223d, log id: 46b44774 >2014-08-04 15:06:49,891 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-84) [53f4cdd5] Initialize Irs proxy from vds: 10.34.62.208 >2014-08-04 15:06:49,899 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-84) [53f4cdd5] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208). >2014-08-04 15:06:49,901 WARN [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-8-thread-44) Executing a command: java.util.concurrent.FutureTask , but note that there are 1 tasks in the queue. >2014-08-04 15:06:49,911 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-44) [53f4cdd5] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 22699016 >2014-08-04 15:06:49,943 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-44) [53f4cdd5] -- executeIrsBrokerCommand: Attempting on storage pool 00000002-0002-0002-0002-0000000001d9 >2014-08-04 15:06:49,945 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-44) [53f4cdd5] START, HSMGetAllTasksInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 33a670c >2014-08-04 15:06:49,950 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-44) [53f4cdd5] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 33a670c >2014-08-04 15:06:49,951 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-44) [53f4cdd5] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 22699016 >2014-08-04 15:06:49,952 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (org.ovirt.thread.pool-8-thread-44) [53f4cdd5] Discovered no tasks on Storage Pool Default >2014-08-04 15:23:56,006 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-31) Principal admin::internal synchronized >2014-08-04 15:24:24,543 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-20) [367a2664] Attempting to update VMs/Templates Ovf. >2014-08-04 16:00:59,087 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp--127.0.0.1-8702-10) Running command: LoginUserCommand internal: false. >2014-08-04 16:00:59,092 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-10) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-08-04 16:00:59,100 INFO [org.ovirt.engine.core.bll.LogoutBySessionCommand] (ajp--127.0.0.1-8702-10) [74f9591] Running command: LogoutBySessionCommand internal: false. >2014-08-04 16:00:59,102 INFO [org.ovirt.engine.core.bll.LogoutUserCommand] (ajp--127.0.0.1-8702-10) [2249ac9f] Running command: LogoutUserCommand internal: false. >2014-08-04 16:00:59,107 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-10) [2249ac9f] Correlation ID: 2249ac9f, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-08-04 16:01:53,134 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp--127.0.0.1-8702-5) Running command: LoginUserCommand internal: false. >2014-08-04 16:01:53,140 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-5) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-08-04 16:01:53,146 INFO [org.ovirt.engine.core.bll.LogoutBySessionCommand] (ajp--127.0.0.1-8702-5) [78671431] Running command: LogoutBySessionCommand internal: false. >2014-08-04 16:01:53,148 INFO [org.ovirt.engine.core.bll.LogoutUserCommand] (ajp--127.0.0.1-8702-5) [54cb46ae] Running command: LogoutUserCommand internal: false. >2014-08-04 16:01:53,152 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-5) [54cb46ae] Correlation ID: 54cb46ae, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-08-04 16:03:44,994 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp--127.0.0.1-8702-4) Running command: LoginUserCommand internal: false. >2014-08-04 16:03:44,998 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-4) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-08-04 16:03:45,006 INFO [org.ovirt.engine.core.bll.LogoutBySessionCommand] (ajp--127.0.0.1-8702-4) [1a52fe7c] Running command: LogoutBySessionCommand internal: false. >2014-08-04 16:03:45,010 INFO [org.ovirt.engine.core.bll.LogoutUserCommand] (ajp--127.0.0.1-8702-4) [20d8892c] Running command: LogoutUserCommand internal: false. >2014-08-04 16:03:45,015 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-4) [20d8892c] Correlation ID: 20d8892c, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-08-04 16:09:21,502 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp--127.0.0.1-8702-7) Running command: LoginUserCommand internal: false. >2014-08-04 16:09:21,508 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-7) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-08-04 16:09:21,518 INFO [org.ovirt.engine.core.bll.LogoutBySessionCommand] (ajp--127.0.0.1-8702-7) [127e1cff] Running command: LogoutBySessionCommand internal: false. >2014-08-04 16:09:21,523 INFO [org.ovirt.engine.core.bll.LogoutUserCommand] (ajp--127.0.0.1-8702-7) [1a57a014] Running command: LogoutUserCommand internal: false. >2014-08-04 16:09:21,529 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-7) [1a57a014] Correlation ID: 1a57a014, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-08-04 16:15:03,561 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp--127.0.0.1-8702-1) Running command: LoginUserCommand internal: false. >2014-08-04 16:15:03,571 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-1) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-08-04 16:15:03,578 INFO [org.ovirt.engine.core.bll.LogoutBySessionCommand] (ajp--127.0.0.1-8702-1) [2b0e536a] Running command: LogoutBySessionCommand internal: false. >2014-08-04 16:15:03,581 INFO [org.ovirt.engine.core.bll.LogoutUserCommand] (ajp--127.0.0.1-8702-1) [10322835] Running command: LogoutUserCommand internal: false. >2014-08-04 16:15:03,585 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-1) [10322835] Correlation ID: 10322835, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-08-04 16:23:56,011 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-100) [3b9a9fa6] Principal admin::internal synchronized >2014-08-04 16:24:24,557 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-9) [aa1055d] Attempting to update VMs/Templates Ovf. >2014-08-04 16:30:46,035 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp--127.0.0.1-8702-11) Running command: LoginUserCommand internal: false. >2014-08-04 16:30:46,053 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-11) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-08-04 16:30:46,059 INFO [org.ovirt.engine.core.bll.LogoutBySessionCommand] (ajp--127.0.0.1-8702-11) [7e1ba738] Running command: LogoutBySessionCommand internal: false. >2014-08-04 16:30:46,062 INFO [org.ovirt.engine.core.bll.LogoutUserCommand] (ajp--127.0.0.1-8702-11) [581cece6] Running command: LogoutUserCommand internal: false. >2014-08-04 16:30:46,066 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-11) [581cece6] Correlation ID: 581cece6, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-08-04 16:33:09,626 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp--127.0.0.1-8702-2) Running command: LoginUserCommand internal: false. >2014-08-04 16:33:09,631 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-2) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-08-04 16:33:09,637 INFO [org.ovirt.engine.core.bll.LogoutBySessionCommand] (ajp--127.0.0.1-8702-2) [43e1cb92] Running command: LogoutBySessionCommand internal: false. >2014-08-04 16:33:09,639 INFO [org.ovirt.engine.core.bll.LogoutUserCommand] (ajp--127.0.0.1-8702-2) [6e7a474d] Running command: LogoutUserCommand internal: false. >2014-08-04 16:33:09,643 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-2) [6e7a474d] Correlation ID: 6e7a474d, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-08-04 17:20:09,784 WARN [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-8-thread-33) Executing a command: java.util.concurrent.FutureTask , but note that there are 1 tasks in the queue. >2014-08-04 17:23:56,016 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-41) [5d8b5376] Principal admin::internal synchronized >2014-08-04 17:24:24,571 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-20) [367a2664] Attempting to update VMs/Templates Ovf. >2014-08-04 17:39:46,542 WARN [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-8-thread-33) Executing a command: java.util.concurrent.FutureTask , but note that there are 1 tasks in the queue. >2014-08-04 18:06:35,729 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-33) domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 in problem. vds: rack >2014-08-04 18:06:35,731 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-33) domain 44559ec0-6bef-491c-bbd6-e67278f37987:export in problem. vds: rack >2014-08-04 18:08:06,328 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-15) [39225a1e] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2014-08-04 18:08:06,332 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-15) [1ecc4e7f] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-08-04 18:08:06,336 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-15) [1ecc4e7f] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Active to Unknown, reason : null >2014-08-04 18:08:06,339 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-15) [1ecc4e7f] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Active to Unknown, reason : null >2014-08-04 18:08:06,354 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-15) [1ecc4e7f] Correlation ID: 1ecc4e7f, Call Stack: null, Custom Event ID: -1, Message: Invalid status on Data Center Default. Setting Data Center status to Non Responsive (On host rack, Error: Cannot access master storage domain). >2014-08-04 18:10:06,379 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-15) [1ecc4e7f] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2014-08-04 18:10:06,381 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-15) [1ecc4e7f] hostFromVds::selectedVds - rack, spmStatus returned null! >2014-08-04 18:10:06,381 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-15) [1ecc4e7f] IrsBroker::Failed::GetStoragePoolInfoVDS >2014-08-04 18:10:06,382 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-15) [1ecc4e7f] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2014-08-04 18:10:06,403 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-7) [35e84c4a] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage >2014-08-04 18:10:06,415 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-7) [35e84c4a] START, ResetIrsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, ignoreStopFailed = true), log id: 4caf530b >2014-08-04 18:10:06,417 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-7) [35e84c4a] START, SpmStopVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9), log id: 249b0ca4 >2014-08-04 18:10:06,422 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-7) [35e84c4a] SpmStopVDSCommand::Stopping SPM on vds rack, pool id 00000002-0002-0002-0002-0000000001d9 >2014-08-04 18:12:06,354 WARN [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-7) [35e84c4a] could not stop spm of pool 00000002-0002-0002-0002-0000000001d9 on vds 17f7fe56-5f32-40ef-b098-611b84484e0b - reason: org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcRunTimeException: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException >2014-08-04 18:12:06,356 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-7) [35e84c4a] FINISH, SpmStopVDSCommand, log id: 249b0ca4 >2014-08-04 18:12:06,359 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-74) [5972dc2] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: VDSNetworkException: org.apache.commons.httpclient.NoHttpResponseException: The server 10.34.62.208 failed to respond >2014-08-04 18:12:06,361 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-74) [5972dc2] hostFromVds::selectedVds - rack, spmStatus returned null! >2014-08-04 18:12:06,361 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ListVDSCommand] (DefaultQuartzScheduler_Worker-69) [52d379dc] Command ListVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-08-04 18:12:06,364 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-69) [52d379dc] Host rack is not responding. It will stay in Connecting state for a grace period of $80 seconds and after that an attempt to fence the host will be issued. >2014-08-04 18:12:06,365 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-69) [52d379dc] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:406) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.fetchRunningVms(VdsUpdateRunTimeInfo.java:971) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVmStats(VdsUpdateRunTimeInfo.java:920) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:638) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:235) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) [:1.7.0_55] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_55] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_55] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_55] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_55] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_55] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_55] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_55] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy82.list(Unknown Source) > at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source) [:1.7.0_55] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_55] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_55] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_55] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_55] > >2014-08-04 18:12:06,372 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-7) [35e84c4a] FINISH, ResetIrsVDSCommand, log id: 4caf530b >2014-08-04 18:12:06,397 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-69) [52d379dc] Failed to refresh VDS , vds = 17f7fe56-5f32-40ef-b098-611b84484e0b : rack, VDS Network Error, continuing. >java.net.ConnectException: Connection refused >2014-08-04 18:12:06,400 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-7) [35e84c4a] Correlation ID: 35e84c4a, Job ID: 96d93662-6798-4e77-a3cc-2c3aebc9afeb, Call Stack: null, Custom Event ID: -1, Message: Failed to Reconstruct Master Domain for Data Center Default. >2014-08-04 18:12:06,404 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-27) [1ecc4e7f] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue >2014-08-04 18:12:06,406 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-49) starting processDomainRecovery for domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 >2014-08-04 18:12:06,457 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-49) Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 was reported by all hosts in status UP as problematic. Not moving the domain to NonOperational because it is being reconstructed now. >2014-08-04 18:12:06,474 WARN [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-49) [67143c69] CanDoAction of action ReconstructMasterDomain failed. Reasons:VAR__ACTION__RECONSTRUCT_MASTER,VAR__TYPE__STORAGE__DOMAIN,ACTION_TYPE_FAILED_NO_VDS_IN_POOL >2014-08-04 18:12:06,482 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-49) [67143c69] Removing vds [] from the domain in maintenance cache >2014-08-04 18:12:06,482 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-49) [67143c69] Removing host(s) [] from hosts unseen domain report cache >2014-08-04 18:12:06,483 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-27) [1ecc4e7f] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue >2014-08-04 18:12:06,485 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-4) starting processDomainRecovery for domain 44559ec0-6bef-491c-bbd6-e67278f37987:export >2014-08-04 18:12:06,489 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-4) Domain 44559ec0-6bef-491c-bbd6-e67278f37987:export was reported by all hosts in status UP as problematic. Moving the domain to NonOperational. >2014-08-04 18:12:06,504 INFO [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-4) [26119ccb] Lock Acquired to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE >, sharedLocks= ] >2014-08-04 18:12:06,510 INFO [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-4) [26119ccb] Running command: DeactivateStorageDomainCommand internal: true. Entities affected : ID: 44559ec0-6bef-491c-bbd6-e67278f37987 Type: Storage >2014-08-04 18:12:06,515 INFO [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-4) [26119ccb] Lock freed to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE >, sharedLocks= ] >2014-08-04 18:12:06,525 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-4) [26119ccb] Correlation ID: 26119ccb, Job ID: 6103c3b6-ac84-41a3-8cf9-2d27d69a53f3, Call Stack: null, Custom Event ID: -1, Message: Storage Domain export (Data Center Default) was deactivated by system because it's not visible by any of the hosts. >2014-08-04 18:12:06,531 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-4) [26119ccb] Removing vds [] from the domain in maintenance cache >2014-08-04 18:12:06,532 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-4) [26119ccb] Removing host(s) [] from hosts unseen domain report cache >2014-08-04 18:12:09,674 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-21) [81f884c] START, GetHardwareInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]), log id: f4310ef >2014-08-04 18:12:09,727 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-21) [81f884c] FINISH, GetHardwareInfoVDSCommand, log id: f4310ef >2014-08-04 18:12:09,752 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-21) [f66741c] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS >2014-08-04 18:12:09,801 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-21) [41c4a42f] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-08-04 18:12:09,809 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-21) [1ccec212] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-08-04 18:12:09,815 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-21) [1ccec212] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, storageType = NFS, connectionList = [{ id: 05047c9a-152e-4ef2-b118-4a1753c1eae0, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35export, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 42f51dff >2014-08-04 18:12:09,872 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-21) [1ccec212] FINISH, ConnectStorageServerVDSCommand, return: {5083f370-af09-457b-9b73-3440e8b65a07=0, 05047c9a-152e-4ef2-b118-4a1753c1eae0=0}, log id: 42f51dff >2014-08-04 18:12:09,873 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-21) [1ccec212] Host rack storage connection was succeeded >2014-08-04 18:12:09,879 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-38) START, ConnectStoragePoolVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, masterVersion = 1), log id: 2117c1e9 >2014-08-04 18:12:16,371 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-36) [3e2ffa8] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Inactive to Unknown, reason : no reporting hosts >2014-08-04 18:12:16,375 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-36) [3e2ffa8] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Inactive to Unknown, reason : no reporting hosts >2014-08-04 18:13:10,272 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-38) FINISH, ConnectStoragePoolVDSCommand, log id: 2117c1e9 >2014-08-04 18:13:10,285 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-21) [1ccec212] START, SetMOMPolicyParametersVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 45bc50ff >2014-08-04 18:13:10,379 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-21) [1ccec212] FINISH, SetMOMPolicyParametersVDSCommand, log id: 45bc50ff >2014-08-04 18:13:10,389 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-21) [1ccec212] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host rack was set to Up. >2014-08-04 18:13:10,395 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-21) [1ccec212] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host rack. >2014-08-04 18:13:10,411 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-21) [675f1edc] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS >2014-08-04 18:13:16,516 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-63) hostFromVds::selectedVds - rack, spmStatus Free, storage pool Default >2014-08-04 18:13:16,522 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-63) starting spm on vds rack, storage pool Default, prevId -1, LVER -1 >2014-08-04 18:13:16,524 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-63) START, SpmStartVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, prevId=-1, prevLVER=-1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 78b5e37d >2014-08-04 18:13:16,573 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-63) spmStart polling started: taskId = ee383937-c5ce-4a40-bbfe-c58e7ec18c06 >2014-08-04 18:13:17,578 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-63) spmStart polling ended: taskId = ee383937-c5ce-4a40-bbfe-c58e7ec18c06 task status = finished >2014-08-04 18:13:17,633 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-63) spmStart polling ended, spm status: SPM >2014-08-04 18:13:17,635 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-63) START, HSMClearTaskVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, taskId=ee383937-c5ce-4a40-bbfe-c58e7ec18c06), log id: 2ab3b892 >2014-08-04 18:13:17,680 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-63) FINISH, HSMClearTaskVDSCommand, log id: 2ab3b892 >2014-08-04 18:13:17,681 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-63) FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@14f2f5c8, log id: 78b5e37d >2014-08-04 18:13:17,685 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-63) Initialize Irs proxy from vds: 10.34.62.208 >2014-08-04 18:13:17,692 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-63) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208). >2014-08-04 18:13:17,695 WARN [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-8-thread-40) Executing a command: java.util.concurrent.FutureTask , but note that there are 1 tasks in the queue. >2014-08-04 18:13:17,706 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-40) START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: f5984ea >2014-08-04 18:13:17,733 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-40) -- executeIrsBrokerCommand: Attempting on storage pool 00000002-0002-0002-0002-0000000001d9 >2014-08-04 18:13:17,735 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-40) START, HSMGetAllTasksInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 1b18ff31 >2014-08-04 18:13:17,740 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-40) FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 1b18ff31 >2014-08-04 18:13:17,741 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-40) FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: f5984ea >2014-08-04 18:13:17,742 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (org.ovirt.thread.pool-8-thread-40) Discovered no tasks on Storage Pool Default >2014-08-04 18:23:56,022 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-76) Principal admin::internal synchronized >2014-08-04 18:24:24,606 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-8) [782c75f] Attempting to update VMs/Templates Ovf. >2014-08-04 19:17:44,691 WARN [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-8-thread-46) Executing a command: java.util.concurrent.FutureTask , but note that there are 1 tasks in the queue. >2014-08-04 19:23:56,027 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-56) [65d9de58] Principal admin::internal synchronized >2014-08-04 19:24:24,623 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-69) [52d379dc] Attempting to update VMs/Templates Ovf. >2014-08-04 20:23:56,033 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-86) [6452f878] Principal admin::internal synchronized >2014-08-04 20:24:24,641 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-8) [782c75f] Attempting to update VMs/Templates Ovf. >2014-08-04 21:12:55,259 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-46) domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 in problem. vds: rack >2014-08-04 21:13:10,409 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-10) domain 44559ec0-6bef-491c-bbd6-e67278f37987:export in problem. vds: rack >2014-08-04 21:14:31,543 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-22) [7e0971f4] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2014-08-04 21:14:31,551 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-22) [4e09115c] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-08-04 21:14:31,557 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-22) [4e09115c] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Active to Unknown, reason : null >2014-08-04 21:14:31,562 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-22) [4e09115c] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Active to Unknown, reason : null >2014-08-04 21:14:31,577 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-22) [4e09115c] Correlation ID: 4e09115c, Call Stack: null, Custom Event ID: -1, Message: Invalid status on Data Center Default. Setting Data Center status to Non Responsive (On host rack, Error: Cannot access master storage domain). >2014-08-04 21:16:31,595 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-22) [4e09115c] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2014-08-04 21:16:31,597 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-22) [4e09115c] hostFromVds::selectedVds - rack, spmStatus returned null! >2014-08-04 21:16:31,598 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-22) [4e09115c] IrsBroker::Failed::GetStoragePoolInfoVDS >2014-08-04 21:16:31,599 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-22) [4e09115c] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2014-08-04 21:16:31,643 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-50) [5e32ac81] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage >2014-08-04 21:16:31,663 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-50) [5e32ac81] START, ResetIrsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, ignoreStopFailed = true), log id: 77d7eddf >2014-08-04 21:16:31,667 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-50) [5e32ac81] START, SpmStopVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9), log id: 3435d519 >2014-08-04 21:16:31,673 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-50) [5e32ac81] SpmStopVDSCommand::Stopping SPM on vds rack, pool id 00000002-0002-0002-0002-0000000001d9 >2014-08-04 21:18:31,567 WARN [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-50) [5e32ac81] could not stop spm of pool 00000002-0002-0002-0002-0000000001d9 on vds 17f7fe56-5f32-40ef-b098-611b84484e0b - reason: org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcRunTimeException: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException >2014-08-04 21:18:31,570 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetStatsVDSCommand] (DefaultQuartzScheduler_Worker-71) [6cd4f56d] Command GetStatsVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-08-04 21:18:31,571 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-49) [72c79e4e] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: VDSNetworkException: org.apache.commons.httpclient.NoHttpResponseException: The server 10.34.62.208 failed to respond >2014-08-04 21:18:31,575 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-49) [72c79e4e] hostFromVds::selectedVds - rack, spmStatus returned null! >2014-08-04 21:18:31,571 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-50) [5e32ac81] FINISH, SpmStopVDSCommand, log id: 3435d519 >2014-08-04 21:18:31,574 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-71) [6cd4f56d] Host rack is not responding. It will stay in Connecting state for a grace period of $80 seconds and after that an attempt to fence the host will be issued. >2014-08-04 21:18:31,579 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-71) [6cd4f56d] vds::refreshVdsStats Failed getVdsStats, vds = 17f7fe56-5f32-40ef-b098-611b84484e0b : rack, error = VDSNetworkException: java.net.ConnectException: Connection refused >2014-08-04 21:18:31,581 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-50) [5e32ac81] FINISH, ResetIrsVDSCommand, log id: 77d7eddf >2014-08-04 21:18:31,581 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-71) [6cd4f56d] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsStats(VdsUpdateRunTimeInfo.java:669) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:623) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:235) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) [:1.7.0_55] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_55] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_55] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_55] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_55] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_55] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_55] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_55] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy82.getVdsStats(Unknown Source) > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_55] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_55] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_55] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_55] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_55] > >2014-08-04 21:18:31,591 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-50) [5e32ac81] Correlation ID: 5e32ac81, Job ID: a6f6dadd-f506-4c1b-8739-9fc8e47a5f3b, Call Stack: null, Custom Event ID: -1, Message: Failed to Reconstruct Master Domain for Data Center Default. >2014-08-04 21:18:31,614 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-71) [6cd4f56d] Failed to refresh VDS , vds = 17f7fe56-5f32-40ef-b098-611b84484e0b : rack, VDS Network Error, continuing. >java.net.ConnectException: Connection refused >2014-08-04 21:18:31,615 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-42) [4e09115c] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue >2014-08-04 21:18:31,618 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-45) starting processDomainRecovery for domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 >2014-08-04 21:18:31,683 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-45) Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 was reported by all hosts in status UP as problematic. Not moving the domain to NonOperational because it is being reconstructed now. >2014-08-04 21:18:31,702 WARN [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-45) [efb44de] CanDoAction of action ReconstructMasterDomain failed. Reasons:VAR__ACTION__RECONSTRUCT_MASTER,VAR__TYPE__STORAGE__DOMAIN,ACTION_TYPE_FAILED_NO_VDS_IN_POOL >2014-08-04 21:18:31,710 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-45) [efb44de] Removing vds [] from the domain in maintenance cache >2014-08-04 21:18:31,710 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-45) [efb44de] Removing host(s) [] from hosts unseen domain report cache >2014-08-04 21:18:31,711 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-42) [4e09115c] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue >2014-08-04 21:18:31,713 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-24) starting processDomainRecovery for domain 44559ec0-6bef-491c-bbd6-e67278f37987:export >2014-08-04 21:18:31,718 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-24) Domain 44559ec0-6bef-491c-bbd6-e67278f37987:export was reported by all hosts in status UP as problematic. Moving the domain to NonOperational. >2014-08-04 21:18:31,733 INFO [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-24) [44482845] Lock Acquired to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE >, sharedLocks= ] >2014-08-04 21:18:31,740 INFO [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-24) [44482845] Running command: DeactivateStorageDomainCommand internal: true. Entities affected : ID: 44559ec0-6bef-491c-bbd6-e67278f37987 Type: Storage >2014-08-04 21:18:31,745 INFO [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-24) [44482845] Lock freed to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE >, sharedLocks= ] >2014-08-04 21:18:31,754 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-24) [44482845] Correlation ID: 44482845, Job ID: 80fcfbd9-a756-47d5-a32a-d0b39d9522b8, Call Stack: null, Custom Event ID: -1, Message: Storage Domain export (Data Center Default) was deactivated by system because it's not visible by any of the hosts. >2014-08-04 21:18:31,759 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-24) [44482845] Removing vds [] from the domain in maintenance cache >2014-08-04 21:18:31,761 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-24) [44482845] Removing host(s) [] from hosts unseen domain report cache >2014-08-04 21:18:34,866 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-55) [7526d482] START, GetHardwareInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]), log id: 3dff2683 >2014-08-04 21:18:34,914 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-55) [7526d482] FINISH, GetHardwareInfoVDSCommand, log id: 3dff2683 >2014-08-04 21:18:34,938 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-55) [a48784f] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS >2014-08-04 21:18:34,988 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-55) [58cb2ded] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-08-04 21:18:34,996 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-55) [73399a21] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-08-04 21:18:35,011 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-55) [73399a21] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, storageType = NFS, connectionList = [{ id: 05047c9a-152e-4ef2-b118-4a1753c1eae0, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35export, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 42dae9a8 >2014-08-04 21:18:35,076 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-55) [73399a21] FINISH, ConnectStorageServerVDSCommand, return: {5083f370-af09-457b-9b73-3440e8b65a07=0, 05047c9a-152e-4ef2-b118-4a1753c1eae0=0}, log id: 42dae9a8 >2014-08-04 21:18:35,078 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-55) [73399a21] Host rack storage connection was succeeded >2014-08-04 21:18:35,087 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-15) START, ConnectStoragePoolVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, masterVersion = 1), log id: 49dbab5f >2014-08-04 21:18:41,580 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-7) [42c2440e] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Inactive to Unknown, reason : no reporting hosts >2014-08-04 21:18:41,584 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-7) [42c2440e] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Inactive to Unknown, reason : no reporting hosts >2014-08-04 21:19:35,537 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-15) FINISH, ConnectStoragePoolVDSCommand, log id: 49dbab5f >2014-08-04 21:19:35,553 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-55) [73399a21] START, SetMOMPolicyParametersVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 168b56d0 >2014-08-04 21:19:35,646 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-55) [73399a21] FINISH, SetMOMPolicyParametersVDSCommand, log id: 168b56d0 >2014-08-04 21:19:35,654 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-55) [73399a21] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host rack was set to Up. >2014-08-04 21:19:35,659 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-55) [73399a21] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host rack. >2014-08-04 21:19:35,674 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-55) [48fc48e5] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS >2014-08-04 21:19:41,710 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-88) hostFromVds::selectedVds - rack, spmStatus Free, storage pool Default >2014-08-04 21:19:41,715 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-88) starting spm on vds rack, storage pool Default, prevId -1, LVER -1 >2014-08-04 21:19:41,717 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-88) START, SpmStartVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, prevId=-1, prevLVER=-1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 1b609664 >2014-08-04 21:19:41,765 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-88) spmStart polling started: taskId = 7ba16e70-2fa1-4939-80c7-bef94b275b97 >2014-08-04 21:19:42,772 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-88) spmStart polling ended: taskId = 7ba16e70-2fa1-4939-80c7-bef94b275b97 task status = finished >2014-08-04 21:19:42,828 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-88) spmStart polling ended, spm status: SPM >2014-08-04 21:19:42,830 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-88) START, HSMClearTaskVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, taskId=7ba16e70-2fa1-4939-80c7-bef94b275b97), log id: 6ce4ec85 >2014-08-04 21:19:42,875 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-88) FINISH, HSMClearTaskVDSCommand, log id: 6ce4ec85 >2014-08-04 21:19:42,876 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-88) FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@269fcb0e, log id: 1b609664 >2014-08-04 21:19:42,880 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-88) Initialize Irs proxy from vds: 10.34.62.208 >2014-08-04 21:19:42,887 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-88) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208). >2014-08-04 21:19:42,889 WARN [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-8-thread-32) Executing a command: java.util.concurrent.FutureTask , but note that there are 1 tasks in the queue. >2014-08-04 21:19:42,899 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-32) START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 75655cc8 >2014-08-04 21:19:42,935 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-32) -- executeIrsBrokerCommand: Attempting on storage pool 00000002-0002-0002-0002-0000000001d9 >2014-08-04 21:19:42,937 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-32) START, HSMGetAllTasksInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 2b804f20 >2014-08-04 21:19:42,942 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-32) FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 2b804f20 >2014-08-04 21:19:42,943 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-32) FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 75655cc8 >2014-08-04 21:19:42,944 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (org.ovirt.thread.pool-8-thread-32) Discovered no tasks on Storage Pool Default >2014-08-04 21:23:56,040 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-80) [386bc8c5] Principal admin::internal synchronized >2014-08-04 21:24:24,655 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-68) [4341002] Attempting to update VMs/Templates Ovf. >2014-08-04 22:23:56,051 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-35) [5b4b3aec] Principal admin::internal synchronized >2014-08-04 22:24:24,668 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-2) Attempting to update VMs/Templates Ovf. >2014-08-04 23:23:56,058 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-80) [386bc8c5] Principal admin::internal synchronized >2014-08-04 23:24:24,701 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-31) Attempting to update VMs/Templates Ovf. >2014-08-05 00:19:30,599 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-16) domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 in problem. vds: rack >2014-08-05 00:19:30,601 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-16) domain 44559ec0-6bef-491c-bbd6-e67278f37987:export in problem. vds: rack >2014-08-05 00:20:59,983 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-37) [68f898a6] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2014-08-05 00:20:59,988 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-37) [10578196] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-08-05 00:20:59,992 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-37) [10578196] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Active to Unknown, reason : null >2014-08-05 00:20:59,994 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-37) [10578196] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Active to Unknown, reason : null >2014-08-05 00:21:00,008 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-37) [10578196] Correlation ID: 10578196, Call Stack: null, Custom Event ID: -1, Message: Invalid status on Data Center Default. Setting Data Center status to Non Responsive (On host rack, Error: Cannot access master storage domain). >2014-08-05 00:23:00,036 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-37) [10578196] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2014-08-05 00:23:00,039 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-37) [10578196] hostFromVds::selectedVds - rack, spmStatus returned null! >2014-08-05 00:23:00,040 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-37) [10578196] IrsBroker::Failed::GetStoragePoolInfoVDS >2014-08-05 00:23:00,041 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-37) [10578196] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2014-08-05 00:23:00,072 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-26) [2bbb80ac] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage >2014-08-05 00:23:00,096 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-26) [2bbb80ac] START, ResetIrsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, ignoreStopFailed = true), log id: 47eb89c9 >2014-08-05 00:23:00,100 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-26) [2bbb80ac] START, SpmStopVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9), log id: 6d337b4f >2014-08-05 00:23:00,106 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-26) [2bbb80ac] SpmStopVDSCommand::Stopping SPM on vds rack, pool id 00000002-0002-0002-0002-0000000001d9 >2014-08-05 00:23:56,063 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-6) [7f1e369e] Principal admin::internal synchronized >2014-08-05 00:24:24,713 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-56) [65d9de58] Attempting to update VMs/Templates Ovf. >2014-08-05 00:25:00,007 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-70) [5f00c7bf] Autorecovering 1 storage domains >2014-08-05 00:25:00,008 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-70) [5f00c7bf] Autorecovering storage domains id: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 >2014-08-05 00:25:00,012 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ListVDSCommand] (DefaultQuartzScheduler_Worker-77) [1f7efd53] Command ListVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-08-05 00:25:00,010 WARN [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-26) [2bbb80ac] could not stop spm of pool 00000002-0002-0002-0002-0000000001d9 on vds 17f7fe56-5f32-40ef-b098-611b84484e0b - reason: org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcRunTimeException: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException >2014-08-05 00:25:00,020 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-77) [1f7efd53] Host rack is not responding. It will stay in Connecting state for a grace period of $80 seconds and after that an attempt to fence the host will be issued. >2014-08-05 00:25:00,019 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-70) [46f87217] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage >2014-08-05 00:25:00,017 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-32) Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: VDSNetworkException: org.apache.commons.httpclient.NoHttpResponseException: The server 10.34.62.208 failed to respond >2014-08-05 00:25:00,026 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-32) hostFromVds::selectedVds - rack, spmStatus returned null! >2014-08-05 00:25:00,026 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-70) [46f87217] ConnectDomainToStorage. Before Connect all hosts to pool. Time:8/5/14 12:25 AM >2014-08-05 00:25:00,023 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-77) [1f7efd53] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:406) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.fetchRunningVms(VdsUpdateRunTimeInfo.java:971) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVmStats(VdsUpdateRunTimeInfo.java:920) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:638) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:235) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) [:1.7.0_55] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_55] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_55] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_55] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_55] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_55] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_55] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_55] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy82.list(Unknown Source) > at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source) [:1.7.0_55] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_55] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_55] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_55] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_55] > >2014-08-05 00:25:00,022 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-26) [2bbb80ac] FINISH, SpmStopVDSCommand, log id: 6d337b4f >2014-08-05 00:25:00,036 INFO [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-8-thread-24) [7b385bda] Running command: ConnectStorageToVdsCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2014-08-05 00:25:00,057 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-24) [7b385bda] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 15d9f1b0 >2014-08-05 00:25:00,058 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-77) [1f7efd53] Failed to refresh VDS , vds = 17f7fe56-5f32-40ef-b098-611b84484e0b : rack, VDS Network Error, continuing. >java.net.ConnectException: Connection refused >2014-08-05 00:25:00,057 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-26) [2bbb80ac] FINISH, ResetIrsVDSCommand, log id: 47eb89c9 >2014-08-05 00:25:00,063 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-24) [7b385bda] Command ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-08-05 00:25:00,066 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-24) [7b385bda] FINISH, ConnectStorageServerVDSCommand, log id: 15d9f1b0 >2014-08-05 00:25:00,068 ERROR [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-8-thread-24) [7b385bda] Command org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-08-05 00:25:00,071 ERROR [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-8-thread-24) [7b385bda] Transaction rolled-back for command: org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand. >2014-08-05 00:25:00,072 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-70) [46f87217] ConnectDomainToStorage. After Connect all hosts to pool. Time:8/5/14 12:25 AM >2014-08-05 00:25:00,076 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-26) [2bbb80ac] Correlation ID: 2bbb80ac, Job ID: 36c08b8e-d5ed-4e31-8f15-324c5c6d989c, Call Stack: null, Custom Event ID: -1, Message: Failed to Reconstruct Master Domain for Data Center Default. >2014-08-05 00:25:00,085 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-23) [10578196] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue >2014-08-05 00:25:00,088 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-16) starting processDomainRecovery for domain 44559ec0-6bef-491c-bbd6-e67278f37987:export >2014-08-05 00:25:00,152 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-16) Domain 44559ec0-6bef-491c-bbd6-e67278f37987:export was reported by all hosts in status UP as problematic. Moving the domain to NonOperational. >2014-08-05 00:25:00,175 INFO [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-16) [4e787b0d] Lock Acquired to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE >, sharedLocks= ] >2014-08-05 00:25:00,182 INFO [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-16) [4e787b0d] Running command: DeactivateStorageDomainCommand internal: true. Entities affected : ID: 44559ec0-6bef-491c-bbd6-e67278f37987 Type: Storage >2014-08-05 00:25:00,188 INFO [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-16) [4e787b0d] Lock freed to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE >, sharedLocks= ] >2014-08-05 00:25:00,198 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-16) [4e787b0d] Correlation ID: 4e787b0d, Job ID: 970b1f0b-ef60-4c89-b93a-4d6589187afd, Call Stack: null, Custom Event ID: -1, Message: Storage Domain export (Data Center Default) was deactivated by system because it's not visible by any of the hosts. >2014-08-05 00:25:00,203 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-16) [4e787b0d] Removing vds [] from the domain in maintenance cache >2014-08-05 00:25:00,204 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-16) [4e787b0d] Removing host(s) [] from hosts unseen domain report cache >2014-08-05 00:25:00,206 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-46) starting processDomainRecovery for domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 >2014-08-05 00:25:00,212 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-46) Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 was reported by all hosts in status UP as problematic. Not moving the domain to NonOperational because it is being reconstructed now. >2014-08-05 00:25:00,230 WARN [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-46) [2f3cf017] CanDoAction of action ReconstructMasterDomain failed. Reasons:VAR__ACTION__RECONSTRUCT_MASTER,VAR__TYPE__STORAGE__DOMAIN,ACTION_TYPE_FAILED_NO_VDS_IN_POOL >2014-08-05 00:25:00,237 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-46) [2f3cf017] Removing vds [] from the domain in maintenance cache >2014-08-05 00:25:00,238 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-46) [2f3cf017] Removing host(s) [] from hosts unseen domain report cache >2014-08-05 00:25:00,239 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-23) [10578196] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue >2014-08-05 00:25:03,131 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-69) [52d379dc] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@78cc51f3 >2014-08-05 00:25:03,131 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-69) [52d379dc] HostName = rack >2014-08-05 00:25:03,132 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-69) [52d379dc] Command GetCapabilitiesVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]) execution failed. Exception: VDSRecoveringException: Recovering from crash or Initializing >2014-08-05 00:25:03,149 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-69) [52d379dc] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSRecoveringException: Recovering from crash or Initializing > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:16) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:566) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:235) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Host rack is initializing. Message: Recovering from crash or Initializing >2014-08-05 00:25:03,153 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-69) [52d379dc] Failed to refresh VDS , vds = 17f7fe56-5f32-40ef-b098-611b84484e0b : rack, error = Recovering from crash or Initializing, continuing. >2014-08-05 00:25:06,386 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-12) START, GetHardwareInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]), log id: 720e86 >2014-08-05 00:25:06,436 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-12) FINISH, GetHardwareInfoVDSCommand, log id: 720e86 >2014-08-05 00:25:06,453 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-12) [439dfcd4] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS >2014-08-05 00:25:06,462 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-12) [439dfcd4] Cancelling the recovery from crash timer for VDS 17f7fe56-5f32-40ef-b098-611b84484e0b because vds started initializing >2014-08-05 00:25:06,463 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-12) [7eb8d6ee] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-08-05 00:25:06,470 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-12) [55f57cce] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-08-05 00:25:06,481 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-12) [55f57cce] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, storageType = NFS, connectionList = [{ id: 05047c9a-152e-4ef2-b118-4a1753c1eae0, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35export, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 2d32f3a7 >2014-08-05 00:25:06,537 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-12) [55f57cce] FINISH, ConnectStorageServerVDSCommand, return: {5083f370-af09-457b-9b73-3440e8b65a07=0, 05047c9a-152e-4ef2-b118-4a1753c1eae0=0}, log id: 2d32f3a7 >2014-08-05 00:25:06,538 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-12) [55f57cce] Host rack storage connection was succeeded >2014-08-05 00:25:06,543 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-49) START, ConnectStoragePoolVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, masterVersion = 1), log id: 652696f9 >2014-08-05 00:25:10,030 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-61) [58bfe349] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Inactive to Unknown, reason : no reporting hosts >2014-08-05 00:25:10,034 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-61) [58bfe349] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Inactive to Unknown, reason : no reporting hosts >2014-08-05 00:26:06,933 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-49) FINISH, ConnectStoragePoolVDSCommand, log id: 652696f9 >2014-08-05 00:26:06,945 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-12) [55f57cce] START, SetMOMPolicyParametersVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 46b6558f >2014-08-05 00:26:07,039 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-12) [55f57cce] FINISH, SetMOMPolicyParametersVDSCommand, log id: 46b6558f >2014-08-05 00:26:07,047 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-12) [55f57cce] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host rack was set to Up. >2014-08-05 00:26:07,055 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-12) [55f57cce] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host rack. >2014-08-05 00:26:07,078 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-12) [7dde64c0] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS >2014-08-05 00:26:10,169 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-13) [519c108f] hostFromVds::selectedVds - rack, spmStatus Free, storage pool Default >2014-08-05 00:26:10,175 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-13) [519c108f] starting spm on vds rack, storage pool Default, prevId -1, LVER -1 >2014-08-05 00:26:10,177 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-13) [519c108f] START, SpmStartVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, prevId=-1, prevLVER=-1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 6f76d94b >2014-08-05 00:26:10,226 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-13) [519c108f] spmStart polling started: taskId = 00e33615-bd15-4363-acff-3b0446834773 >2014-08-05 00:26:11,233 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-13) [519c108f] spmStart polling ended: taskId = 00e33615-bd15-4363-acff-3b0446834773 task status = finished >2014-08-05 00:26:11,292 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-13) [519c108f] spmStart polling ended, spm status: SPM >2014-08-05 00:26:11,294 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-13) [519c108f] START, HSMClearTaskVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, taskId=00e33615-bd15-4363-acff-3b0446834773), log id: 2d9d319e >2014-08-05 00:26:11,339 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-13) [519c108f] FINISH, HSMClearTaskVDSCommand, log id: 2d9d319e >2014-08-05 00:26:11,340 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-13) [519c108f] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@4fbf4b05, log id: 6f76d94b >2014-08-05 00:26:11,344 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-13) [519c108f] Initialize Irs proxy from vds: 10.34.62.208 >2014-08-05 00:26:11,350 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-13) [519c108f] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208). >2014-08-05 00:26:11,361 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-44) [519c108f] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 34abe4f1 >2014-08-05 00:26:11,389 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-44) [519c108f] -- executeIrsBrokerCommand: Attempting on storage pool 00000002-0002-0002-0002-0000000001d9 >2014-08-05 00:26:11,391 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-44) [519c108f] START, HSMGetAllTasksInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 55e32341 >2014-08-05 00:26:11,397 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-44) [519c108f] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 55e32341 >2014-08-05 00:26:11,398 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-44) [519c108f] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 34abe4f1 >2014-08-05 00:26:11,399 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (org.ovirt.thread.pool-8-thread-44) [519c108f] Discovered no tasks on Storage Pool Default >2014-08-05 01:23:56,069 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-75) [6e437c9b] Principal admin::internal synchronized >2014-08-05 01:24:24,715 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-49) [72c79e4e] Attempting to update VMs/Templates Ovf. >2014-08-05 02:23:56,078 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-56) [65d9de58] Principal admin::internal synchronized >2014-08-05 02:24:24,732 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-4) Attempting to update VMs/Templates Ovf. >2014-08-05 03:23:56,084 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-19) Principal admin::internal synchronized >2014-08-05 03:24:24,751 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-45) [22578e65] Attempting to update VMs/Templates Ovf. >2014-08-05 03:26:04,814 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-4) domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 in problem. vds: rack >2014-08-05 03:26:04,816 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-4) domain 44559ec0-6bef-491c-bbd6-e67278f37987:export in problem. vds: rack >2014-08-05 03:27:30,362 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-90) Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2014-08-05 03:27:30,366 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-90) [12692311] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-08-05 03:27:30,370 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-90) [12692311] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Active to Unknown, reason : null >2014-08-05 03:27:30,372 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-90) [12692311] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Active to Unknown, reason : null >2014-08-05 03:27:30,383 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-90) [12692311] Correlation ID: 12692311, Call Stack: null, Custom Event ID: -1, Message: Invalid status on Data Center Default. Setting Data Center status to Non Responsive (On host rack, Error: Cannot access master storage domain). >2014-08-05 03:29:30,426 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-90) [12692311] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2014-08-05 03:29:30,429 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-90) [12692311] hostFromVds::selectedVds - rack, spmStatus returned null! >2014-08-05 03:29:30,430 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-90) [12692311] IrsBroker::Failed::GetStoragePoolInfoVDS >2014-08-05 03:29:30,431 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-90) [12692311] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2014-08-05 03:29:30,475 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-11) [6d21d912] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage >2014-08-05 03:29:30,498 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-11) [6d21d912] START, ResetIrsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, ignoreStopFailed = true), log id: 198fd227 >2014-08-05 03:29:30,501 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-11) [6d21d912] START, SpmStopVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9), log id: 66bea16c >2014-08-05 03:29:30,547 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-11) [6d21d912] SpmStopVDSCommand::Stopping SPM on vds rack, pool id 00000002-0002-0002-0002-0000000001d9 >2014-08-05 03:30:00,005 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-27) Autorecovering 1 storage domains >2014-08-05 03:30:00,006 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-27) Autorecovering storage domains id: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 >2014-08-05 03:30:00,008 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-27) [48cf3129] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage >2014-08-05 03:30:00,009 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-27) [48cf3129] ConnectDomainToStorage. Before Connect all hosts to pool. Time:8/5/14 3:30 AM >2014-08-05 03:30:00,018 INFO [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-8-thread-26) [75294a38] Running command: ConnectStorageToVdsCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2014-08-05 03:30:00,023 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-26) [75294a38] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 562929a4 >2014-08-05 03:31:30,407 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-71) [6cd4f56d] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2014-08-05 03:31:30,409 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-71) [6cd4f56d] hostFromVds::selectedVds - rack, spmStatus returned null! >2014-08-05 03:31:30,410 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-71) [6cd4f56d] IrsBroker::Failed::GetStoragePoolInfoVDS >2014-08-05 03:31:30,411 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-71) [6cd4f56d] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2014-08-05 03:32:30,502 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-26) [75294a38] FINISH, ConnectStorageServerVDSCommand, return: {5083f370-af09-457b-9b73-3440e8b65a07=0}, log id: 562929a4 >2014-08-05 03:32:30,505 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-27) [48cf3129] ConnectDomainToStorage. After Connect all hosts to pool. Time:8/5/14 3:32 AM >2014-08-05 03:32:30,548 WARN [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-11) [6d21d912] could not stop spm of pool 00000002-0002-0002-0002-0000000001d9 on vds 17f7fe56-5f32-40ef-b098-611b84484e0b - reason: org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcRunTimeException: java.util.concurrent.TimeoutException >2014-08-05 03:32:30,551 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-11) [6d21d912] FINISH, SpmStopVDSCommand, log id: 66bea16c >2014-08-05 03:32:30,556 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-11) [6d21d912] FINISH, ResetIrsVDSCommand, log id: 198fd227 >2014-08-05 03:32:30,566 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-11) [6d21d912] Correlation ID: 6d21d912, Job ID: f5df7420-b8d2-4b9c-aa1f-913622b83139, Call Stack: null, Custom Event ID: -1, Message: Failed to Reconstruct Master Domain for Data Center Default. >2014-08-05 03:32:30,573 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-44) [12692311] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue >2014-08-05 03:32:30,575 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-6) starting processDomainRecovery for domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 >2014-08-05 03:32:30,637 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-6) Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 was reported by all hosts in status UP as problematic. Not moving the domain to NonOperational because it is being reconstructed now. >2014-08-05 03:32:30,660 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-6) [2048c5bd] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage >2014-08-05 03:32:30,684 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-6) [2048c5bd] Removing vds [] from the domain in maintenance cache >2014-08-05 03:32:30,685 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-6) [2048c5bd] Removing host(s) [] from hosts unseen domain report cache >2014-08-05 03:32:30,686 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-44) [12692311] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue >2014-08-05 03:32:30,688 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-3) starting processDomainRecovery for domain 44559ec0-6bef-491c-bbd6-e67278f37987:export >2014-08-05 03:32:30,693 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-3) Domain 44559ec0-6bef-491c-bbd6-e67278f37987:export was reported by all hosts in status UP as problematic. Moving the domain to NonOperational. >2014-08-05 03:32:30,710 INFO [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-3) [47b01ade] Lock Acquired to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE >, sharedLocks= ] >2014-08-05 03:32:30,716 INFO [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-3) [47b01ade] Running command: DeactivateStorageDomainCommand internal: true. Entities affected : ID: 44559ec0-6bef-491c-bbd6-e67278f37987 Type: Storage >2014-08-05 03:32:30,728 INFO [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-3) [47b01ade] Lock freed to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE >, sharedLocks= ] >2014-08-05 03:32:30,742 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-3) [47b01ade] Correlation ID: 47b01ade, Job ID: ed978873-7309-4d0f-a618-a58b6deff278, Call Stack: null, Custom Event ID: -1, Message: Storage Domain export (Data Center Default) was deactivated by system because it's not visible by any of the hosts. >2014-08-05 03:32:30,747 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-3) [47b01ade] Removing vds [] from the domain in maintenance cache >2014-08-05 03:32:30,748 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-3) [47b01ade] Removing host(s) [] from hosts unseen domain report cache >2014-08-05 03:33:30,424 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-3) Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: VDSNetworkException: org.apache.commons.httpclient.NoHttpResponseException: The server 10.34.62.208 failed to respond >2014-08-05 03:33:30,426 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-3) hostFromVds::selectedVds - rack, spmStatus returned null! >2014-08-05 03:33:31,177 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ListVDSCommand] (DefaultQuartzScheduler_Worker-84) [53f4cdd5] Command ListVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]) execution failed. Exception: VDSRecoveringException: Recovering from crash or Initializing >2014-08-05 03:33:31,179 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-84) [53f4cdd5] Failed vds listing, vds = 17f7fe56-5f32-40ef-b098-611b84484e0b : rack, error = org.ovirt.engine.core.vdsbroker.vdsbroker.VDSRecoveringException: Recovering from crash or Initializing >2014-08-05 03:33:31,191 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-84) [53f4cdd5] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSRecoveringException: Recovering from crash or Initializing > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.vdsbroker.ListVDSCommand.executeVdsBrokerCommand(ListVDSCommand.java:24) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:406) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.fetchRunningVms(VdsUpdateRunTimeInfo.java:971) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVmStats(VdsUpdateRunTimeInfo.java:920) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:638) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:235) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Host rack is initializing. Message: Recovering from crash or Initializing >2014-08-05 03:33:31,199 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-84) [53f4cdd5] Failed to refresh VDS , vds = 17f7fe56-5f32-40ef-b098-611b84484e0b : rack, error = Recovering from crash or Initializing, continuing. >2014-08-05 03:33:34,432 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-96) START, GetHardwareInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]), log id: 4fdd9f51 >2014-08-05 03:33:34,484 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-96) FINISH, GetHardwareInfoVDSCommand, log id: 4fdd9f51 >2014-08-05 03:33:34,511 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-96) [3f9d9215] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS >2014-08-05 03:33:34,561 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-96) [3f9d9215] Cancelling the recovery from crash timer for VDS 17f7fe56-5f32-40ef-b098-611b84484e0b because vds started initializing >2014-08-05 03:33:34,563 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-96) [5368d9dd] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-08-05 03:33:34,571 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-96) [6941298b] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-08-05 03:33:34,579 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-96) [6941298b] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, storageType = NFS, connectionList = [{ id: 05047c9a-152e-4ef2-b118-4a1753c1eae0, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35export, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 6030bbc6 >2014-08-05 03:33:34,639 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-96) [6941298b] FINISH, ConnectStorageServerVDSCommand, return: {5083f370-af09-457b-9b73-3440e8b65a07=0, 05047c9a-152e-4ef2-b118-4a1753c1eae0=0}, log id: 6030bbc6 >2014-08-05 03:33:34,640 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-96) [6941298b] Host rack storage connection was succeeded >2014-08-05 03:33:34,647 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-35) START, ConnectStoragePoolVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, masterVersion = 1), log id: 85d57cb >2014-08-05 03:33:40,430 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-41) [5d8b5376] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Inactive to Unknown, reason : no reporting hosts >2014-08-05 03:33:40,434 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-41) [5d8b5376] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Inactive to Unknown, reason : no reporting hosts >2014-08-05 03:34:35,051 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-35) FINISH, ConnectStoragePoolVDSCommand, log id: 85d57cb >2014-08-05 03:34:35,066 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-96) [6941298b] START, SetMOMPolicyParametersVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 125596 >2014-08-05 03:34:35,161 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-96) [6941298b] FINISH, SetMOMPolicyParametersVDSCommand, log id: 125596 >2014-08-05 03:34:35,169 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-96) [6941298b] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host rack was set to Up. >2014-08-05 03:34:35,176 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-96) [6941298b] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host rack. >2014-08-05 03:34:35,194 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-96) [3648c2dc] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS >2014-08-05 03:34:40,546 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-51) [4760e42] hostFromVds::selectedVds - rack, spmStatus Free, storage pool Default >2014-08-05 03:34:40,551 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-51) [4760e42] starting spm on vds rack, storage pool Default, prevId -1, LVER -1 >2014-08-05 03:34:40,553 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-51) [4760e42] START, SpmStartVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, prevId=-1, prevLVER=-1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 62c89002 >2014-08-05 03:34:40,601 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-51) [4760e42] spmStart polling started: taskId = 6a20d58f-0e63-4969-b02c-d9f94fcd3f67 >2014-08-05 03:34:41,607 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-51) [4760e42] spmStart polling ended: taskId = 6a20d58f-0e63-4969-b02c-d9f94fcd3f67 task status = finished >2014-08-05 03:34:41,662 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-51) [4760e42] spmStart polling ended, spm status: SPM >2014-08-05 03:34:41,664 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-51) [4760e42] START, HSMClearTaskVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, taskId=6a20d58f-0e63-4969-b02c-d9f94fcd3f67), log id: 65f5ada6 >2014-08-05 03:34:41,710 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-51) [4760e42] FINISH, HSMClearTaskVDSCommand, log id: 65f5ada6 >2014-08-05 03:34:41,712 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-51) [4760e42] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@4f93cf03, log id: 62c89002 >2014-08-05 03:34:41,716 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-51) [4760e42] Initialize Irs proxy from vds: 10.34.62.208 >2014-08-05 03:34:41,724 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-51) [4760e42] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208). >2014-08-05 03:34:41,726 WARN [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-8-thread-28) Executing a command: java.util.concurrent.FutureTask , but note that there are 1 tasks in the queue. >2014-08-05 03:34:41,742 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-28) [4760e42] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 4b74362b >2014-08-05 03:34:41,782 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-28) [4760e42] -- executeIrsBrokerCommand: Attempting on storage pool 00000002-0002-0002-0002-0000000001d9 >2014-08-05 03:34:41,784 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-28) [4760e42] START, HSMGetAllTasksInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 4e792ca4 >2014-08-05 03:34:41,790 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-28) [4760e42] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 4e792ca4 >2014-08-05 03:34:41,791 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-28) [4760e42] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 4b74362b >2014-08-05 03:34:41,792 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (org.ovirt.thread.pool-8-thread-28) [4760e42] Discovered no tasks on Storage Pool Default >2014-08-05 03:35:35,001 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-81) [7125e62c] Start deleteAgedOutAuditLogs >2014-08-05 03:35:35,001 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-26) [6bd9692] Start deleteAgedOutCommandEntities >2014-08-05 03:35:35,007 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-81) [7125e62c] Finished deleteAgedOutAuditLogs >2014-08-05 03:35:35,009 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-26) [6bd9692] Finished deleteAgedOutCommandEntities >2014-08-05 04:23:56,089 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-60) [1cb6c42c] Principal admin::internal synchronized >2014-08-05 04:24:24,772 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-3) Attempting to update VMs/Templates Ovf. >2014-08-05 05:23:56,095 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-19) Principal admin::internal synchronized >2014-08-05 05:24:24,807 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-51) [4760e42] Attempting to update VMs/Templates Ovf. >2014-08-05 06:23:56,102 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-24) [61a1f9b5] Principal admin::internal synchronized >2014-08-05 06:24:24,843 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-2) Attempting to update VMs/Templates Ovf. >2014-08-05 06:34:20,632 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-36) domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 in problem. vds: rack >2014-08-05 06:34:20,634 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-36) domain 44559ec0-6bef-491c-bbd6-e67278f37987:export in problem. vds: rack >2014-08-05 06:35:56,586 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-76) Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2014-08-05 06:35:56,591 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-76) [5b2cf9d8] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-08-05 06:35:56,595 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-76) [5b2cf9d8] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Active to Unknown, reason : null >2014-08-05 06:35:56,598 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-76) [5b2cf9d8] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Active to Unknown, reason : null >2014-08-05 06:35:56,613 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-76) [5b2cf9d8] Correlation ID: 5b2cf9d8, Call Stack: null, Custom Event ID: -1, Message: Invalid status on Data Center Default. Setting Data Center status to Non Responsive (On host rack, Error: Cannot access master storage domain). >2014-08-05 06:37:56,643 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-76) [5b2cf9d8] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2014-08-05 06:37:56,646 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-76) [5b2cf9d8] hostFromVds::selectedVds - rack, spmStatus returned null! >2014-08-05 06:37:56,647 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-76) [5b2cf9d8] IrsBroker::Failed::GetStoragePoolInfoVDS >2014-08-05 06:37:56,648 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-76) [5b2cf9d8] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2014-08-05 06:37:56,693 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-4) [54d106fb] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage >2014-08-05 06:37:56,715 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-4) [54d106fb] START, ResetIrsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, ignoreStopFailed = true), log id: 678172ed >2014-08-05 06:37:56,718 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-4) [54d106fb] START, SpmStopVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9), log id: 2faa05d8 >2014-08-05 06:37:56,724 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-4) [54d106fb] SpmStopVDSCommand::Stopping SPM on vds rack, pool id 00000002-0002-0002-0002-0000000001d9 >2014-08-05 06:39:56,613 WARN [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-4) [54d106fb] could not stop spm of pool 00000002-0002-0002-0002-0000000001d9 on vds 17f7fe56-5f32-40ef-b098-611b84484e0b - reason: org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcRunTimeException: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException >2014-08-05 06:39:56,615 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetStatsVDSCommand] (DefaultQuartzScheduler_Worker-54) [14ab6114] Command GetStatsVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-08-05 06:39:56,614 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-92) Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: VDSNetworkException: org.apache.commons.httpclient.NoHttpResponseException: The server 10.34.62.208 failed to respond >2014-08-05 06:39:56,616 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-4) [54d106fb] FINISH, SpmStopVDSCommand, log id: 2faa05d8 >2014-08-05 06:39:56,621 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-54) [14ab6114] Host rack is not responding. It will stay in Connecting state for a grace period of $80 seconds and after that an attempt to fence the host will be issued. >2014-08-05 06:39:56,623 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-54) [14ab6114] vds::refreshVdsStats Failed getVdsStats, vds = 17f7fe56-5f32-40ef-b098-611b84484e0b : rack, error = VDSNetworkException: java.net.ConnectException: Connection refused >2014-08-05 06:39:56,620 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-92) hostFromVds::selectedVds - rack, spmStatus returned null! >2014-08-05 06:39:56,624 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-54) [14ab6114] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsStats(VdsUpdateRunTimeInfo.java:669) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:623) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:235) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) [:1.7.0_55] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_55] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_55] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_55] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_55] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_55] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_55] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_55] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy82.getVdsStats(Unknown Source) > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_55] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_55] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_55] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_55] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_55] > >2014-08-05 06:39:56,627 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-4) [54d106fb] FINISH, ResetIrsVDSCommand, log id: 678172ed >2014-08-05 06:39:56,656 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-54) [14ab6114] Failed to refresh VDS , vds = 17f7fe56-5f32-40ef-b098-611b84484e0b : rack, VDS Network Error, continuing. >java.net.ConnectException: Connection refused >2014-08-05 06:39:56,663 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-4) [54d106fb] Correlation ID: 54d106fb, Job ID: 972c6800-188a-4a8b-b123-7aa1d39d00a5, Call Stack: null, Custom Event ID: -1, Message: Failed to Reconstruct Master Domain for Data Center Default. >2014-08-05 06:39:56,674 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-12) [5b2cf9d8] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue >2014-08-05 06:39:56,678 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-16) starting processDomainRecovery for domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 >2014-08-05 06:39:56,763 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-16) Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 was reported by all hosts in status UP as problematic. Not moving the domain to NonOperational because it is being reconstructed now. >2014-08-05 06:39:56,780 WARN [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-16) [7043d56e] CanDoAction of action ReconstructMasterDomain failed. Reasons:VAR__ACTION__RECONSTRUCT_MASTER,VAR__TYPE__STORAGE__DOMAIN,ACTION_TYPE_FAILED_NO_VDS_IN_POOL >2014-08-05 06:39:56,787 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-16) [7043d56e] Removing vds [] from the domain in maintenance cache >2014-08-05 06:39:56,788 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-16) [7043d56e] Removing host(s) [] from hosts unseen domain report cache >2014-08-05 06:39:56,789 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-12) [5b2cf9d8] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue >2014-08-05 06:39:56,791 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-45) starting processDomainRecovery for domain 44559ec0-6bef-491c-bbd6-e67278f37987:export >2014-08-05 06:39:56,796 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-45) Domain 44559ec0-6bef-491c-bbd6-e67278f37987:export was reported by all hosts in status UP as problematic. Moving the domain to NonOperational. >2014-08-05 06:39:56,811 INFO [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-45) [6bbc9db9] Lock Acquired to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE >, sharedLocks= ] >2014-08-05 06:39:56,818 INFO [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-45) [6bbc9db9] Running command: DeactivateStorageDomainCommand internal: true. Entities affected : ID: 44559ec0-6bef-491c-bbd6-e67278f37987 Type: Storage >2014-08-05 06:39:56,823 INFO [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-45) [6bbc9db9] Lock freed to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE >, sharedLocks= ] >2014-08-05 06:39:56,833 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-45) [6bbc9db9] Correlation ID: 6bbc9db9, Job ID: ef20d657-e1e2-42a3-a8e8-2540f7918007, Call Stack: null, Custom Event ID: -1, Message: Storage Domain export (Data Center Default) was deactivated by system because it's not visible by any of the hosts. >2014-08-05 06:39:56,838 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-45) [6bbc9db9] Removing vds [] from the domain in maintenance cache >2014-08-05 06:39:56,839 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-45) [6bbc9db9] Removing host(s) [] from hosts unseen domain report cache >2014-08-05 06:39:59,733 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-8) [782c75f] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@3e9f5f42 >2014-08-05 06:39:59,734 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-8) [782c75f] HostName = rack >2014-08-05 06:39:59,735 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-8) [782c75f] Command GetCapabilitiesVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]) execution failed. Exception: VDSRecoveringException: Recovering from crash or Initializing >2014-08-05 06:39:59,748 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-8) [782c75f] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSRecoveringException: Recovering from crash or Initializing > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:16) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:566) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:235) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Host rack is initializing. Message: Recovering from crash or Initializing >2014-08-05 06:39:59,756 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-8) [782c75f] Failed to refresh VDS , vds = 17f7fe56-5f32-40ef-b098-611b84484e0b : rack, error = Recovering from crash or Initializing, continuing. >2014-08-05 06:40:00,007 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-97) [1b0a6877] Autorecovering 2 storage domains >2014-08-05 06:40:00,008 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-97) [1b0a6877] Autorecovering storage domains id: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 >2014-08-05 06:40:00,010 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-97) [150723da] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage >2014-08-05 06:40:00,011 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-97) [150723da] ConnectDomainToStorage. Before Connect all hosts to pool. Time:8/5/14 6:40 AM >2014-08-05 06:40:00,016 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-97) [150723da] ConnectDomainToStorage. After Connect all hosts to pool. Time:8/5/14 6:40 AM >2014-08-05 06:40:00,018 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-97) [150723da] Autorecovering storage domains id: 44559ec0-6bef-491c-bbd6-e67278f37987 >2014-08-05 06:40:00,020 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-97) [5c182324] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 44559ec0-6bef-491c-bbd6-e67278f37987 Type: Storage >2014-08-05 06:40:00,021 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-97) [5c182324] ConnectDomainToStorage. Before Connect all hosts to pool. Time:8/5/14 6:40 AM >2014-08-05 06:40:00,025 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-97) [5c182324] ConnectDomainToStorage. After Connect all hosts to pool. Time:8/5/14 6:40 AM >2014-08-05 06:40:02,991 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-80) [386bc8c5] START, GetHardwareInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]), log id: 6e5bc94a >2014-08-05 06:40:03,041 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-80) [386bc8c5] FINISH, GetHardwareInfoVDSCommand, log id: 6e5bc94a >2014-08-05 06:40:03,058 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-80) [31903ec5] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS >2014-08-05 06:40:03,104 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-80) [31903ec5] Cancelling the recovery from crash timer for VDS 17f7fe56-5f32-40ef-b098-611b84484e0b because vds started initializing >2014-08-05 06:40:03,105 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-80) [7ebb35ee] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-08-05 06:40:03,111 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-80) [4f16934] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-08-05 06:40:03,117 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-80) [4f16934] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, storageType = NFS, connectionList = [{ id: 05047c9a-152e-4ef2-b118-4a1753c1eae0, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35export, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 4c92140a >2014-08-05 06:40:03,175 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-80) [4f16934] FINISH, ConnectStorageServerVDSCommand, return: {5083f370-af09-457b-9b73-3440e8b65a07=0, 05047c9a-152e-4ef2-b118-4a1753c1eae0=0}, log id: 4c92140a >2014-08-05 06:40:03,177 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-80) [4f16934] Host rack storage connection was succeeded >2014-08-05 06:40:03,191 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-30) START, ConnectStoragePoolVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, masterVersion = 1), log id: 7162ee48 >2014-08-05 06:40:06,628 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-34) [36ef1be6] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Inactive to Unknown, reason : no reporting hosts >2014-08-05 06:40:06,633 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-34) [36ef1be6] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Inactive to Unknown, reason : no reporting hosts >2014-08-05 06:41:03,607 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-30) FINISH, ConnectStoragePoolVDSCommand, log id: 7162ee48 >2014-08-05 06:41:03,620 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-80) [4f16934] START, SetMOMPolicyParametersVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 347ae785 >2014-08-05 06:41:03,714 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-80) [4f16934] FINISH, SetMOMPolicyParametersVDSCommand, log id: 347ae785 >2014-08-05 06:41:03,723 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-80) [4f16934] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host rack was set to Up. >2014-08-05 06:41:03,731 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-80) [4f16934] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host rack. >2014-08-05 06:41:03,751 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-80) [16f4cea2] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS >2014-08-05 06:41:06,748 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-53) [3c2dfb77] hostFromVds::selectedVds - rack, spmStatus Free, storage pool Default >2014-08-05 06:41:06,755 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-53) [3c2dfb77] starting spm on vds rack, storage pool Default, prevId -1, LVER -1 >2014-08-05 06:41:06,759 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-53) [3c2dfb77] START, SpmStartVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, prevId=-1, prevLVER=-1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 74393759 >2014-08-05 06:41:06,810 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-53) [3c2dfb77] spmStart polling started: taskId = 56799788-dd0e-46b4-955b-3a4cd8bdb866 >2014-08-05 06:41:07,820 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-53) [3c2dfb77] spmStart polling ended: taskId = 56799788-dd0e-46b4-955b-3a4cd8bdb866 task status = finished >2014-08-05 06:41:07,877 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-53) [3c2dfb77] spmStart polling ended, spm status: SPM >2014-08-05 06:41:07,879 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-53) [3c2dfb77] START, HSMClearTaskVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, taskId=56799788-dd0e-46b4-955b-3a4cd8bdb866), log id: 1b7df594 >2014-08-05 06:41:07,924 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-53) [3c2dfb77] FINISH, HSMClearTaskVDSCommand, log id: 1b7df594 >2014-08-05 06:41:07,925 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-53) [3c2dfb77] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@206d3049, log id: 74393759 >2014-08-05 06:41:07,929 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-53) [3c2dfb77] Initialize Irs proxy from vds: 10.34.62.208 >2014-08-05 06:41:07,936 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-53) [3c2dfb77] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208). >2014-08-05 06:41:07,938 WARN [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-8-thread-40) Executing a command: java.util.concurrent.FutureTask , but note that there are 1 tasks in the queue. >2014-08-05 06:41:07,949 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-40) [3c2dfb77] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 7761496 >2014-08-05 06:41:07,975 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-40) [3c2dfb77] -- executeIrsBrokerCommand: Attempting on storage pool 00000002-0002-0002-0002-0000000001d9 >2014-08-05 06:41:07,976 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-40) [3c2dfb77] START, HSMGetAllTasksInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 291e2a13 >2014-08-05 06:41:07,982 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-40) [3c2dfb77] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 291e2a13 >2014-08-05 06:41:07,983 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-40) [3c2dfb77] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 7761496 >2014-08-05 06:41:07,985 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (org.ovirt.thread.pool-8-thread-40) [3c2dfb77] Discovered no tasks on Storage Pool Default >2014-08-05 06:54:52,529 WARN [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-8-thread-37) Executing a command: java.util.concurrent.FutureTask , but note that there are 1 tasks in the queue. >2014-08-05 07:15:59,741 WARN [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-8-thread-8) Executing a command: java.util.concurrent.FutureTask , but note that there are 1 tasks in the queue. >2014-08-05 07:23:56,107 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-95) [6a286cf6] Principal admin::internal synchronized >2014-08-05 07:24:24,877 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-33) [1f9421b1] Attempting to update VMs/Templates Ovf. >2014-08-05 08:23:56,115 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-51) [4760e42] Principal admin::internal synchronized >2014-08-05 08:24:24,899 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-76) [5b2cf9d8] Attempting to update VMs/Templates Ovf. >2014-08-05 09:23:56,125 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-83) Principal admin::internal synchronized >2014-08-05 09:24:24,913 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-27) [48cf3129] Attempting to update VMs/Templates Ovf. >2014-08-05 09:40:55,390 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-45) domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 in problem. vds: rack >2014-08-05 09:40:55,392 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-45) domain 44559ec0-6bef-491c-bbd6-e67278f37987:export in problem. vds: rack >2014-08-05 09:42:27,398 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-12) [7dde64c0] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2014-08-05 09:42:27,402 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-12) [5b5ccf8] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-08-05 09:42:27,406 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-12) [5b5ccf8] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Active to Unknown, reason : null >2014-08-05 09:42:27,408 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-12) [5b5ccf8] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Active to Unknown, reason : null >2014-08-05 09:42:27,420 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-12) [5b5ccf8] Correlation ID: 5b5ccf8, Call Stack: null, Custom Event ID: -1, Message: Invalid status on Data Center Default. Setting Data Center status to Non Responsive (On host rack, Error: Cannot access master storage domain). >2014-08-05 09:44:27,461 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-12) [5b5ccf8] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2014-08-05 09:44:27,463 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-12) [5b5ccf8] hostFromVds::selectedVds - rack, spmStatus returned null! >2014-08-05 09:44:27,464 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-12) [5b5ccf8] IrsBroker::Failed::GetStoragePoolInfoVDS >2014-08-05 09:44:27,465 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-12) [5b5ccf8] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2014-08-05 09:44:27,508 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-44) [2897c8c0] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage >2014-08-05 09:44:27,529 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-44) [2897c8c0] START, ResetIrsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, ignoreStopFailed = true), log id: 79a4d674 >2014-08-05 09:44:27,533 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-44) [2897c8c0] START, SpmStopVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9), log id: 4ee75edb >2014-08-05 09:44:27,539 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-44) [2897c8c0] SpmStopVDSCommand::Stopping SPM on vds rack, pool id 00000002-0002-0002-0002-0000000001d9 >2014-08-05 09:45:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-68) [4341002] Autorecovering 1 storage domains >2014-08-05 09:45:00,005 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-68) [4341002] Autorecovering storage domains id: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 >2014-08-05 09:45:00,007 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-68) [64321ee4] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage >2014-08-05 09:45:00,009 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-68) [64321ee4] ConnectDomainToStorage. Before Connect all hosts to pool. Time:8/5/14 9:45 AM >2014-08-05 09:45:00,020 INFO [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-8-thread-40) [45ab4934] Running command: ConnectStorageToVdsCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2014-08-05 09:45:00,025 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-40) [45ab4934] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 4a6166bf >2014-08-05 09:46:27,431 WARN [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-44) [2897c8c0] could not stop spm of pool 00000002-0002-0002-0002-0000000001d9 on vds 17f7fe56-5f32-40ef-b098-611b84484e0b - reason: org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcRunTimeException: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException >2014-08-05 09:46:27,443 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-84) [53f4cdd5] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: VDSNetworkException: org.apache.commons.httpclient.NoHttpResponseException: The server 10.34.62.208 failed to respond >2014-08-05 09:46:27,449 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-44) [2897c8c0] FINISH, SpmStopVDSCommand, log id: 4ee75edb >2014-08-05 09:46:27,442 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ListVDSCommand] (DefaultQuartzScheduler_Worker-83) Command ListVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-08-05 09:46:27,441 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-40) [45ab4934] Command ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-08-05 09:46:27,464 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-83) Host rack is not responding. It will stay in Connecting state for a grace period of $80 seconds and after that an attempt to fence the host will be issued. >2014-08-05 09:46:27,471 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-44) [2897c8c0] FINISH, ResetIrsVDSCommand, log id: 79a4d674 >2014-08-05 09:46:27,451 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-84) [53f4cdd5] hostFromVds::selectedVds - rack, spmStatus returned null! >2014-08-05 09:46:27,475 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-83) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:406) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.fetchRunningVms(VdsUpdateRunTimeInfo.java:971) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVmStats(VdsUpdateRunTimeInfo.java:920) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:638) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:235) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) [:1.7.0_55] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_55] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_55] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_55] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_55] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_55] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_55] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_55] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy82.list(Unknown Source) > at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source) [:1.7.0_55] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_55] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_55] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_55] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_55] > >2014-08-05 09:46:27,489 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-44) [2897c8c0] Correlation ID: 2897c8c0, Job ID: edacbacd-2001-482d-b262-7b2810e9c972, Call Stack: null, Custom Event ID: -1, Message: Failed to Reconstruct Master Domain for Data Center Default. >2014-08-05 09:46:27,480 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-40) [45ab4934] FINISH, ConnectStorageServerVDSCommand, log id: 4a6166bf >2014-08-05 09:46:27,511 ERROR [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-8-thread-40) [45ab4934] Command org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-08-05 09:46:27,523 ERROR [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-8-thread-40) [45ab4934] Transaction rolled-back for command: org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand. >2014-08-05 09:46:27,525 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-68) [64321ee4] ConnectDomainToStorage. After Connect all hosts to pool. Time:8/5/14 9:46 AM >2014-08-05 09:46:27,526 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-22) [5b5ccf8] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue >2014-08-05 09:46:27,528 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-38) starting processDomainRecovery for domain 44559ec0-6bef-491c-bbd6-e67278f37987:export >2014-08-05 09:46:27,538 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-83) Failed to refresh VDS , vds = 17f7fe56-5f32-40ef-b098-611b84484e0b : rack, VDS Network Error, continuing. >java.net.ConnectException: Connection refused >2014-08-05 09:46:27,590 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-38) Domain 44559ec0-6bef-491c-bbd6-e67278f37987:export was reported by all hosts in status UP as problematic. Moving the domain to NonOperational. >2014-08-05 09:46:27,606 INFO [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-38) [18dc9b9c] Lock Acquired to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE >, sharedLocks= ] >2014-08-05 09:46:27,613 INFO [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-38) [18dc9b9c] Running command: DeactivateStorageDomainCommand internal: true. Entities affected : ID: 44559ec0-6bef-491c-bbd6-e67278f37987 Type: Storage >2014-08-05 09:46:27,618 INFO [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-38) [18dc9b9c] Lock freed to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE >, sharedLocks= ] >2014-08-05 09:46:27,626 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-38) [18dc9b9c] Correlation ID: 18dc9b9c, Job ID: b0afea17-65c6-41e8-a7db-8287a9099f43, Call Stack: null, Custom Event ID: -1, Message: Storage Domain export (Data Center Default) was deactivated by system because it's not visible by any of the hosts. >2014-08-05 09:46:27,632 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-38) [18dc9b9c] Removing vds [] from the domain in maintenance cache >2014-08-05 09:46:27,632 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-38) [18dc9b9c] Removing host(s) [] from hosts unseen domain report cache >2014-08-05 09:46:27,634 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-4) starting processDomainRecovery for domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 >2014-08-05 09:46:27,639 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-4) Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 was reported by all hosts in status UP as problematic. Not moving the domain to NonOperational because it is being reconstructed now. >2014-08-05 09:46:27,662 WARN [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-4) [4340f131] CanDoAction of action ReconstructMasterDomain failed. Reasons:VAR__ACTION__RECONSTRUCT_MASTER,VAR__TYPE__STORAGE__DOMAIN,ACTION_TYPE_FAILED_NO_VDS_IN_POOL >2014-08-05 09:46:27,669 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-4) [4340f131] Removing vds [] from the domain in maintenance cache >2014-08-05 09:46:27,669 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-4) [4340f131] Removing host(s) [] from hosts unseen domain report cache >2014-08-05 09:46:27,670 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-22) [5b5ccf8] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue >2014-08-05 09:46:30,704 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-4) Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@1fb31abe >2014-08-05 09:46:30,706 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-4) HostName = rack >2014-08-05 09:46:30,706 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-4) Command GetCapabilitiesVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]) execution failed. Exception: VDSRecoveringException: Recovering from crash or Initializing >2014-08-05 09:46:30,726 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-4) Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSRecoveringException: Recovering from crash or Initializing > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:16) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:566) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:235) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Host rack is initializing. Message: Recovering from crash or Initializing >2014-08-05 09:46:30,734 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-4) Failed to refresh VDS , vds = 17f7fe56-5f32-40ef-b098-611b84484e0b : rack, error = Recovering from crash or Initializing, continuing. >2014-08-05 09:46:33,984 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-86) [6452f878] START, GetHardwareInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]), log id: 363f0058 >2014-08-05 09:46:34,035 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-86) [6452f878] FINISH, GetHardwareInfoVDSCommand, log id: 363f0058 >2014-08-05 09:46:34,090 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-86) [4a9cb1ef] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS >2014-08-05 09:46:34,100 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-86) [4a9cb1ef] Cancelling the recovery from crash timer for VDS 17f7fe56-5f32-40ef-b098-611b84484e0b because vds started initializing >2014-08-05 09:46:34,102 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-86) [655d9eeb] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-08-05 09:46:34,116 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-86) [4e3e768c] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-08-05 09:46:34,141 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-86) [4e3e768c] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, storageType = NFS, connectionList = [{ id: 05047c9a-152e-4ef2-b118-4a1753c1eae0, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35export, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 70acfc72 >2014-08-05 09:46:34,164 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-86) [4e3e768c] FINISH, ConnectStorageServerVDSCommand, return: {5083f370-af09-457b-9b73-3440e8b65a07=0, 05047c9a-152e-4ef2-b118-4a1753c1eae0=0}, log id: 70acfc72 >2014-08-05 09:46:34,165 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-86) [4e3e768c] Host rack storage connection was succeeded >2014-08-05 09:46:34,172 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-6) START, ConnectStoragePoolVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, masterVersion = 1), log id: 232fb377 >2014-08-05 09:46:37,487 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-91) [64566e87] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Inactive to Unknown, reason : no reporting hosts >2014-08-05 09:46:37,492 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-91) [64566e87] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Inactive to Unknown, reason : no reporting hosts >2014-08-05 09:47:34,568 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-6) FINISH, ConnectStoragePoolVDSCommand, log id: 232fb377 >2014-08-05 09:47:34,587 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-86) [4e3e768c] START, SetMOMPolicyParametersVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 3d7ac75b >2014-08-05 09:47:34,679 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-86) [4e3e768c] FINISH, SetMOMPolicyParametersVDSCommand, log id: 3d7ac75b >2014-08-05 09:47:34,689 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-86) [4e3e768c] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host rack was set to Up. >2014-08-05 09:47:34,697 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-86) [4e3e768c] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host rack. >2014-08-05 09:47:34,725 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-86) [10e8de77] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS >2014-08-05 09:47:37,688 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-14) hostFromVds::selectedVds - rack, spmStatus Free, storage pool Default >2014-08-05 09:47:37,695 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-14) starting spm on vds rack, storage pool Default, prevId -1, LVER -1 >2014-08-05 09:47:37,698 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-14) START, SpmStartVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, prevId=-1, prevLVER=-1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 6c6254d8 >2014-08-05 09:47:37,747 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-14) spmStart polling started: taskId = 01c1a364-29b3-4511-bc23-3ab6d412a182 >2014-08-05 09:47:38,756 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-14) spmStart polling ended: taskId = 01c1a364-29b3-4511-bc23-3ab6d412a182 task status = finished >2014-08-05 09:47:38,812 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-14) spmStart polling ended, spm status: SPM >2014-08-05 09:47:38,815 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-14) START, HSMClearTaskVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, taskId=01c1a364-29b3-4511-bc23-3ab6d412a182), log id: 43382406 >2014-08-05 09:47:38,861 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-14) FINISH, HSMClearTaskVDSCommand, log id: 43382406 >2014-08-05 09:47:38,862 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-14) FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@1bf8b515, log id: 6c6254d8 >2014-08-05 09:47:38,866 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-14) Initialize Irs proxy from vds: 10.34.62.208 >2014-08-05 09:47:38,872 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-14) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208). >2014-08-05 09:47:38,873 WARN [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-8-thread-10) Executing a command: java.util.concurrent.FutureTask , but note that there are 1 tasks in the queue. >2014-08-05 09:47:38,884 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-10) START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: de30a73 >2014-08-05 09:47:38,911 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-10) -- executeIrsBrokerCommand: Attempting on storage pool 00000002-0002-0002-0002-0000000001d9 >2014-08-05 09:47:38,913 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-10) START, HSMGetAllTasksInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 3131d937 >2014-08-05 09:47:38,918 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-10) FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 3131d937 >2014-08-05 09:47:38,919 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-10) FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: de30a73 >2014-08-05 09:47:38,920 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (org.ovirt.thread.pool-8-thread-10) Discovered no tasks on Storage Pool Default >2014-08-05 10:23:56,134 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-27) [48cf3129] Principal admin::internal synchronized >2014-08-05 10:24:24,928 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-89) [1b83a377] Attempting to update VMs/Templates Ovf. >2014-08-05 10:58:07,282 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp--127.0.0.1-8702-3) Running command: LoginUserCommand internal: false. >2014-08-05 10:58:07,290 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-3) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-08-05 10:58:07,306 INFO [org.ovirt.engine.core.bll.LogoutBySessionCommand] (ajp--127.0.0.1-8702-3) [66c1e4ac] Running command: LogoutBySessionCommand internal: false. >2014-08-05 10:58:07,311 INFO [org.ovirt.engine.core.bll.LogoutUserCommand] (ajp--127.0.0.1-8702-3) [44a1e617] Running command: LogoutUserCommand internal: false. >2014-08-05 10:58:07,318 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-3) [44a1e617] Correlation ID: 44a1e617, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-08-05 10:58:35,535 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp--127.0.0.1-8702-10) Running command: LoginUserCommand internal: false. >2014-08-05 10:58:35,539 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-10) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-08-05 10:58:35,549 INFO [org.ovirt.engine.core.bll.LogoutBySessionCommand] (ajp--127.0.0.1-8702-10) [5119ae73] Running command: LogoutBySessionCommand internal: false. >2014-08-05 10:58:35,552 INFO [org.ovirt.engine.core.bll.LogoutUserCommand] (ajp--127.0.0.1-8702-10) [4a7eb4b7] Running command: LogoutUserCommand internal: false. >2014-08-05 10:58:35,556 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-10) [4a7eb4b7] Correlation ID: 4a7eb4b7, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-08-05 10:58:40,989 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp--127.0.0.1-8702-5) Running command: LoginUserCommand internal: false. >2014-08-05 10:58:40,993 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-5) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-08-05 10:58:41,463 INFO [org.ovirt.engine.core.bll.storage.AddEmptyStoragePoolCommand] (ajp--127.0.0.1-8702-1) [50420e3c-b64b-4342_addDataCenter_create] Running command: AddEmptyStoragePoolCommand internal: false. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2014-08-05 10:58:41,491 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-1) [50420e3c-b64b-4342_addDataCenter_create] Correlation ID: 50420e3c-b64b-4342_addDataCenter_create, Call Stack: null, Custom Event ID: -1, Message: Data Center datacenter_None, Compatibility Version 3.4 and Quota Type DISABLED was added by admin >2014-08-05 10:58:43,788 INFO [org.ovirt.engine.core.bll.AddVdsGroupCommand] (ajp--127.0.0.1-8702-4) [bcf22733-13b6-486e_addCluster_create] Running command: AddVdsGroupCommand internal: false. Entities affected : ID: b69d184e-0a37-4ed1-b097-86e0d2968149 Type: StoragePool >2014-08-05 10:58:43,854 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-4) [bcf22733-13b6-486e_addCluster_create] Correlation ID: bcf22733-13b6-486e_addCluster_create, Call Stack: null, Custom Event ID: -1, Message: Cluster cluster_None was added by admin >2014-08-05 10:58:50,990 ERROR [org.ovirt.engine.core.bll.AddVdsCommand] (ajp--127.0.0.1-8702-10) [c8d39794-ba0b-4cf5_addHost_create] Failed to authenticate session with host slot-7.rhev.lab.eng.brq.redhat.com: javax.naming.AuthenticationException: SSH authentication to 'root@10.34.63.137' failed. Please verify provided credentials. Make sure host is configured for password authentication > at org.ovirt.engine.core.uutils.ssh.SSHClient.authenticate(SSHClient.java:365) [uutils.jar:] > at org.ovirt.engine.core.bll.AddVdsCommand.canConnect(AddVdsCommand.java:430) [bll.jar:] > at org.ovirt.engine.core.bll.AddVdsCommand.canDoAction(AddVdsCommand.java:342) [bll.jar:] > at org.ovirt.engine.core.bll.CommandBase.internalCanDoAction(CommandBase.java:737) [bll.jar:] > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:339) [bll.jar:] > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:430) [bll.jar:] > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:411) [bll.jar:] > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:369) [bll.jar:] > at sun.reflect.GeneratedMethodAccessor391.invoke(Unknown Source) [:1.7.0_55] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55] > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) [jboss-as-ee-7.1.1.Final.jar:7.1.1.Final] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final] > at org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:374) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final] > at org.jboss.as.weld.ejb.Jsr299BindingsInterceptor.delegateInterception(Jsr299BindingsInterceptor.java:114) [jboss-as-weld-7.1.1.Final.jar:7.1.1.Final] > at org.jboss.as.weld.ejb.Jsr299BindingsInterceptor.doMethodInterception(Jsr299BindingsInterceptor.java:125) [jboss-as-weld-7.1.1.Final.jar:7.1.1.Final] > at org.jboss.as.weld.ejb.Jsr299BindingsInterceptor.processInvocation(Jsr299BindingsInterceptor.java:135) [jboss-as-weld-7.1.1.Final.jar:7.1.1.Final] > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:36) [jboss-as-ee-7.1.1.Final.jar:7.1.1.Final] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final] > at org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:374) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final] > at org.ovirt.engine.core.bll.interceptors.ThreadLocalSessionCleanerInterceptor.injectWebContextToThreadLocal(ThreadLocalSessionCleanerInterceptor.java:13) [bll.jar:] > at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source) [:1.7.0_55] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55] > at org.jboss.as.ee.component.ManagedReferenceLifecycleMethodInterceptorFactory$ManagedReferenceLifecycleMethodInterceptor.processInvocation(ManagedReferenceLifecycleMethodInterceptorFactory.java:123) [jboss-as-ee-7.1.1.Final.jar:7.1.1.Final] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final] > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final] > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:36) [jboss-as-ee-7.1.1.Final.jar:7.1.1.Final] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final] > at org.jboss.as.weld.ejb.EjbRequestScopeActivationInterceptor.processInvocation(EjbRequestScopeActivationInterceptor.java:82) [jboss-as-weld-7.1.1.Final.jar:7.1.1.Final] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final] > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final] > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final] > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) [jboss-as-ee-7.1.1.Final.jar:7.1.1.Final] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final] > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:53) [jboss-as-ejb3-7.1.1.Final.jar:7.1.1.Final] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final] > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:211) [jboss-as-ejb3-7.1.1.Final.jar:7.1.1.Final] > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:363) [jboss-as-ejb3-7.1.1.Final.jar:7.1.1.Final] > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:194) [jboss-as-ejb3-7.1.1.Final.jar:7.1.1.Final] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final] > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) [jboss-as-ejb3-7.1.1.Final.jar:7.1.1.Final] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final] > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) [jboss-as-ejb3-7.1.1.Final.jar:7.1.1.Final] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final] > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) [jboss-as-ee-7.1.1.Final.jar:7.1.1.Final] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final] > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) [jboss-as-ee-7.1.1.Final.jar:7.1.1.Final] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final] > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final] > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) [jboss-as-ee-7.1.1.Final.jar:7.1.1.Final] > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:173) [jboss-as-ee-7.1.1.Final.jar:7.1.1.Final] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final] > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final] > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) [jboss-as-ee-7.1.1.Final.jar:7.1.1.Final] > at org.ovirt.engine.core.common.interfaces.BackendLocal$$$view7.runAction(Unknown Source) [common.jar:] > at org.ovirt.engine.api.restapi.resource.BackendResource.doAction(BackendResource.java:259) > at org.ovirt.engine.api.restapi.resource.AbstractBackendCollectionResource.doCreateEntity(AbstractBackendCollectionResource.java:243) > at org.ovirt.engine.api.restapi.resource.AbstractBackendCollectionResource.performCreate(AbstractBackendCollectionResource.java:120) > at org.ovirt.engine.api.restapi.resource.AbstractBackendCollectionResource.performCreate(AbstractBackendCollectionResource.java:110) > at org.ovirt.engine.api.restapi.resource.AbstractBackendCollectionResource.performCreate(AbstractBackendCollectionResource.java:129) > at org.ovirt.engine.api.restapi.resource.BackendHostsResource.add(BackendHostsResource.java:100) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_55] > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_55] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55] > at org.jboss.resteasy.core.MethodInjectorImpl.invoke(MethodInjectorImpl.java:155) [resteasy-jaxrs-2.3.2.Final.jar:] > at org.jboss.resteasy.core.ResourceMethod.invokeOnTarget(ResourceMethod.java:257) [resteasy-jaxrs-2.3.2.Final.jar:] > at org.jboss.resteasy.core.ResourceMethod.invoke(ResourceMethod.java:222) [resteasy-jaxrs-2.3.2.Final.jar:] > at org.jboss.resteasy.core.ResourceMethod.invoke(ResourceMethod.java:211) [resteasy-jaxrs-2.3.2.Final.jar:] > at org.jboss.resteasy.core.SynchronousDispatcher.getResponse(SynchronousDispatcher.java:525) [resteasy-jaxrs-2.3.2.Final.jar:] > at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:502) [resteasy-jaxrs-2.3.2.Final.jar:] > at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:119) [resteasy-jaxrs-2.3.2.Final.jar:] > at org.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher.service(ServletContainerDispatcher.java:208) [resteasy-jaxrs-2.3.2.Final.jar:] > at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:55) [resteasy-jaxrs-2.3.2.Final.jar:] > at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:50) [resteasy-jaxrs-2.3.2.Final.jar:] > at javax.servlet.http.HttpServlet.service(HttpServlet.java:847) [jboss-servlet-api_3.0_spec-1.0.0.Final.jar:1.0.0.Final] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:329) > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) > at org.ovirt.engine.core.aaa.filters.RestApiSessionMgmtFilter.doFilter(RestApiSessionMgmtFilter.java:69) [aaa.jar:] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) > at org.ovirt.engine.core.aaa.filters.EnforceAuthFilter.doFilter(EnforceAuthFilter.java:39) [aaa.jar:] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) > at org.ovirt.engine.core.aaa.filters.LoginFilter.doFilter(LoginFilter.java:73) [aaa.jar:] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) > at org.ovirt.engine.core.aaa.filters.NegotiationFilter.doFilter(NegotiationFilter.java:105) [aaa.jar:] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) > at org.ovirt.engine.core.aaa.filters.BasicAuthenticationFilter.doFilter(BasicAuthenticationFilter.java:75) [aaa.jar:] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) > at org.ovirt.engine.core.aaa.filters.SessionValidationFilter.doFilter(SessionValidationFilter.java:63) [aaa.jar:] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) > at org.ovirt.engine.core.aaa.filters.RestApiSessionValidationFilter.doFilter(RestApiSessionValidationFilter.java:31) [aaa.jar:] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) > at org.ovirt.engine.api.common.security.CSRFProtectionFilter.doFilter(CSRFProtectionFilter.java:110) > at org.ovirt.engine.api.common.security.CSRFProtectionFilter.doFilter(CSRFProtectionFilter.java:101) > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) > at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:275) > at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:161) > at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:489) > at org.jboss.as.web.security.SecurityContextAssociationValve.invoke(SecurityContextAssociationValve.java:153) > at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:155) > at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102) > at org.jboss.web.rewrite.RewriteValve.invoke(RewriteValve.java:466) > at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109) > at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:368) > at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:505) > at org.apache.coyote.ajp.AjpProtocol$AjpConnectionHandler.process(AjpProtocol.java:445) > at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:930) > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_55] > >2014-08-05 10:58:51,100 WARN [org.ovirt.engine.core.bll.AddVdsCommand] (ajp--127.0.0.1-8702-10) [c8d39794-ba0b-4cf5_addHost_create] CanDoAction of action AddVds failed. Reasons:VAR__ACTION__ADD,VAR__TYPE__HOST,$server 10.34.63.137,VDS_CANNOT_AUTHENTICATE_TO_SERVER >2014-08-05 10:58:51,119 ERROR [org.ovirt.engine.api.restapi.resource.AbstractBackendResource] (ajp--127.0.0.1-8702-10) Operation Failed: [Cannot add Host. SSH authentication failed, verify authentication parameters are correct (Username/Password, public-key etc.) You may refer to the engine.log file for further details.] >2014-08-05 11:23:56,140 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-44) [29bdffa0] Principal admin::internal synchronized >2014-08-05 11:24:24,967 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-88) Attempting to update VMs/Templates Ovf. >2014-08-05 12:23:56,150 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-49) [72c79e4e] Principal admin::internal synchronized >2014-08-05 12:24:24,996 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-31) Attempting to update VMs/Templates Ovf. >2014-08-05 12:47:18,129 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-44) domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 in problem. vds: rack >2014-08-05 12:47:33,260 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-42) domain 44559ec0-6bef-491c-bbd6-e67278f37987:export in problem. vds: rack >2014-08-05 12:48:58,026 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-68) [64321ee4] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2014-08-05 12:48:58,031 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-68) [5d663c2e] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-08-05 12:48:58,035 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-68) [5d663c2e] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Active to Unknown, reason : null >2014-08-05 12:48:58,037 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-68) [5d663c2e] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Active to Unknown, reason : null >2014-08-05 12:48:58,052 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-68) [5d663c2e] Correlation ID: 5d663c2e, Call Stack: null, Custom Event ID: -1, Message: Invalid status on Data Center Default. Setting Data Center status to Non Responsive (On host rack, Error: Cannot access master storage domain). >2014-08-05 12:50:58,077 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-68) [5d663c2e] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2014-08-05 12:50:58,079 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-68) [5d663c2e] hostFromVds::selectedVds - rack, spmStatus returned null! >2014-08-05 12:50:58,080 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-68) [5d663c2e] IrsBroker::Failed::GetStoragePoolInfoVDS >2014-08-05 12:50:58,081 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-68) [5d663c2e] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2014-08-05 12:50:58,117 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-28) [66d0e553] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage >2014-08-05 12:50:58,136 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-28) [66d0e553] START, ResetIrsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, ignoreStopFailed = true), log id: 5eba1a6f >2014-08-05 12:50:58,140 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-28) [66d0e553] START, SpmStopVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9), log id: 45b2da5a >2014-08-05 12:50:58,146 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-28) [66d0e553] SpmStopVDSCommand::Stopping SPM on vds rack, pool id 00000002-0002-0002-0002-0000000001d9 >2014-08-05 12:52:58,047 WARN [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-28) [66d0e553] could not stop spm of pool 00000002-0002-0002-0002-0000000001d9 on vds 17f7fe56-5f32-40ef-b098-611b84484e0b - reason: org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcRunTimeException: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException >2014-08-05 12:52:58,052 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-60) [1cb6c42c] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: VDSNetworkException: org.apache.commons.httpclient.NoHttpResponseException: The server 10.34.62.208 failed to respond >2014-08-05 12:52:58,051 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ListVDSCommand] (DefaultQuartzScheduler_Worker-63) Command ListVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-08-05 12:52:58,054 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-28) [66d0e553] FINISH, SpmStopVDSCommand, log id: 45b2da5a >2014-08-05 12:52:58,058 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-60) [1cb6c42c] hostFromVds::selectedVds - rack, spmStatus returned null! >2014-08-05 12:52:58,058 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-63) Host rack is not responding. It will stay in Connecting state for a grace period of $80 seconds and after that an attempt to fence the host will be issued. >2014-08-05 12:52:58,065 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-28) [66d0e553] FINISH, ResetIrsVDSCommand, log id: 5eba1a6f >2014-08-05 12:52:58,063 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-63) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:406) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.fetchRunningVms(VdsUpdateRunTimeInfo.java:971) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVmStats(VdsUpdateRunTimeInfo.java:920) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:638) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:235) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) [:1.7.0_55] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_55] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_55] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_55] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_55] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_55] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_55] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_55] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy82.list(Unknown Source) > at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source) [:1.7.0_55] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_55] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_55] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_55] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_55] > >2014-08-05 12:52:58,081 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-28) [66d0e553] Correlation ID: 66d0e553, Job ID: 7d1d8dac-1268-418e-b3a0-6b5ff20b3f54, Call Stack: null, Custom Event ID: -1, Message: Failed to Reconstruct Master Domain for Data Center Default. >2014-08-05 12:52:58,126 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-11) [5d663c2e] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue >2014-08-05 12:52:58,130 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-33) starting processDomainRecovery for domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 >2014-08-05 12:52:58,131 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-63) Failed to refresh VDS , vds = 17f7fe56-5f32-40ef-b098-611b84484e0b : rack, VDS Network Error, continuing. >java.net.ConnectException: Connection refused >2014-08-05 12:52:58,185 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-33) Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 was reported by all hosts in status UP as problematic. Not moving the domain to NonOperational because it is being reconstructed now. >2014-08-05 12:52:58,202 WARN [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-33) [fe90986] CanDoAction of action ReconstructMasterDomain failed. Reasons:VAR__ACTION__RECONSTRUCT_MASTER,VAR__TYPE__STORAGE__DOMAIN,ACTION_TYPE_FAILED_NO_VDS_IN_POOL >2014-08-05 12:52:58,209 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-33) [fe90986] Removing vds [] from the domain in maintenance cache >2014-08-05 12:52:58,210 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-33) [fe90986] Removing host(s) [] from hosts unseen domain report cache >2014-08-05 12:52:58,211 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-11) [5d663c2e] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue >2014-08-05 12:52:58,213 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-1) starting processDomainRecovery for domain 44559ec0-6bef-491c-bbd6-e67278f37987:export >2014-08-05 12:52:58,217 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-1) Domain 44559ec0-6bef-491c-bbd6-e67278f37987:export was reported by all hosts in status UP as problematic. Moving the domain to NonOperational. >2014-08-05 12:52:58,231 INFO [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-1) [36e57e40] Lock Acquired to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE >, sharedLocks= ] >2014-08-05 12:52:58,238 INFO [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-1) [36e57e40] Running command: DeactivateStorageDomainCommand internal: true. Entities affected : ID: 44559ec0-6bef-491c-bbd6-e67278f37987 Type: Storage >2014-08-05 12:52:58,243 INFO [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-1) [36e57e40] Lock freed to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE >, sharedLocks= ] >2014-08-05 12:52:58,254 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-1) [36e57e40] Correlation ID: 36e57e40, Job ID: f6e6aa62-cc55-43a0-91e7-c8f1b1ba6157, Call Stack: null, Custom Event ID: -1, Message: Storage Domain export (Data Center Default) was deactivated by system because it's not visible by any of the hosts. >2014-08-05 12:52:58,259 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-1) [36e57e40] Removing vds [] from the domain in maintenance cache >2014-08-05 12:52:58,260 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-1) [36e57e40] Removing host(s) [] from hosts unseen domain report cache >2014-08-05 12:53:01,401 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-54) [14ab6114] START, GetHardwareInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]), log id: be2058e >2014-08-05 12:53:01,456 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-54) [14ab6114] FINISH, GetHardwareInfoVDSCommand, log id: be2058e >2014-08-05 12:53:01,480 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-54) [577a9eb9] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS >2014-08-05 12:53:01,529 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-54) [28b9964a] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-08-05 12:53:01,537 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-54) [30c8b1ab] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-08-05 12:53:01,551 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-54) [30c8b1ab] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, storageType = NFS, connectionList = [{ id: 05047c9a-152e-4ef2-b118-4a1753c1eae0, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35export, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 5b881afd >2014-08-05 12:53:01,614 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-54) [30c8b1ab] FINISH, ConnectStorageServerVDSCommand, return: {5083f370-af09-457b-9b73-3440e8b65a07=0, 05047c9a-152e-4ef2-b118-4a1753c1eae0=0}, log id: 5b881afd >2014-08-05 12:53:01,615 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-54) [30c8b1ab] Host rack storage connection was succeeded >2014-08-05 12:53:01,624 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-45) START, ConnectStoragePoolVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, masterVersion = 1), log id: 24a0d4ce >2014-08-05 12:53:08,065 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-81) [7125e62c] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Inactive to Unknown, reason : no reporting hosts >2014-08-05 12:53:08,069 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-81) [7125e62c] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Inactive to Unknown, reason : no reporting hosts >2014-08-05 12:54:02,056 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-45) FINISH, ConnectStoragePoolVDSCommand, log id: 24a0d4ce >2014-08-05 12:54:02,069 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-54) [30c8b1ab] START, SetMOMPolicyParametersVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 70aa1a64 >2014-08-05 12:54:02,162 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-54) [30c8b1ab] FINISH, SetMOMPolicyParametersVDSCommand, log id: 70aa1a64 >2014-08-05 12:54:02,171 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-54) [30c8b1ab] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host rack was set to Up. >2014-08-05 12:54:02,177 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-54) [30c8b1ab] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host rack. >2014-08-05 12:54:02,196 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-54) [23c334ab] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS >2014-08-05 12:54:08,180 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-40) hostFromVds::selectedVds - rack, spmStatus Free, storage pool Default >2014-08-05 12:54:08,185 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-40) starting spm on vds rack, storage pool Default, prevId -1, LVER -1 >2014-08-05 12:54:08,187 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-40) START, SpmStartVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, prevId=-1, prevLVER=-1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 30de20c3 >2014-08-05 12:54:08,237 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-40) spmStart polling started: taskId = cf641d74-96d5-4d43-80a1-b258419f6a9b >2014-08-05 12:54:09,257 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-40) spmStart polling ended: taskId = cf641d74-96d5-4d43-80a1-b258419f6a9b task status = finished >2014-08-05 12:54:09,315 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-40) spmStart polling ended, spm status: SPM >2014-08-05 12:54:09,318 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-40) START, HSMClearTaskVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, taskId=cf641d74-96d5-4d43-80a1-b258419f6a9b), log id: 20002368 >2014-08-05 12:54:09,362 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-40) FINISH, HSMClearTaskVDSCommand, log id: 20002368 >2014-08-05 12:54:09,364 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-40) FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@68e40257, log id: 30de20c3 >2014-08-05 12:54:09,368 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-40) Initialize Irs proxy from vds: 10.34.62.208 >2014-08-05 12:54:09,375 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-40) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208). >2014-08-05 12:54:09,376 WARN [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-8-thread-6) Executing a command: java.util.concurrent.FutureTask , but note that there are 1 tasks in the queue. >2014-08-05 12:54:09,391 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-6) START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 7ac06cba >2014-08-05 12:54:09,427 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-6) -- executeIrsBrokerCommand: Attempting on storage pool 00000002-0002-0002-0002-0000000001d9 >2014-08-05 12:54:09,429 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-6) START, HSMGetAllTasksInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 4d85319e >2014-08-05 12:54:09,434 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-6) FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 4d85319e >2014-08-05 12:54:09,435 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-6) FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 7ac06cba >2014-08-05 12:54:09,436 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (org.ovirt.thread.pool-8-thread-6) Discovered no tasks on Storage Pool Default >2014-08-05 13:18:03,125 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp--127.0.0.1-8702-11) Running command: LoginUserCommand internal: false. >2014-08-05 13:18:03,130 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-11) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-08-05 13:18:03,138 INFO [org.ovirt.engine.core.bll.LogoutBySessionCommand] (ajp--127.0.0.1-8702-11) [7f717a80] Running command: LogoutBySessionCommand internal: false. >2014-08-05 13:18:03,142 INFO [org.ovirt.engine.core.bll.LogoutUserCommand] (ajp--127.0.0.1-8702-11) [64313665] Running command: LogoutUserCommand internal: false. >2014-08-05 13:18:03,146 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-11) [64313665] Correlation ID: 64313665, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-08-05 13:23:56,157 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-11) [aeb0141] Principal admin::internal synchronized >2014-08-05 13:24:25,028 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-4) Attempting to update VMs/Templates Ovf. >2014-08-05 14:23:56,164 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-54) [23c334ab] Principal admin::internal synchronized >2014-08-05 14:24:25,046 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-18) [5497a121] Attempting to update VMs/Templates Ovf. >2014-08-05 15:23:56,170 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-61) [58bfe349] Principal admin::internal synchronized >2014-08-05 15:24:25,080 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-77) [1f7efd53] Attempting to update VMs/Templates Ovf. >2014-08-05 15:58:54,530 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-50) domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 in problem. vds: rack >2014-08-05 15:58:54,532 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-50) domain 44559ec0-6bef-491c-bbd6-e67278f37987:export in problem. vds: rack >2014-08-05 16:01:35,473 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-82) [3d60c3e4] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2014-08-05 16:01:35,476 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-82) [3e9bda56] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-08-05 16:01:35,478 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-82) [3e9bda56] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Active to Unknown, reason : null >2014-08-05 16:01:35,480 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-82) [3e9bda56] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Active to Unknown, reason : null >2014-08-05 16:01:35,490 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-82) [3e9bda56] Correlation ID: 3e9bda56, Call Stack: null, Custom Event ID: -1, Message: Invalid status on Data Center Default. Setting Data Center status to Non Responsive (On host rack, Error: Cannot access master storage domain). >2014-08-05 16:03:35,529 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-82) [3e9bda56] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2014-08-05 16:03:35,531 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-82) [3e9bda56] hostFromVds::selectedVds - rack, spmStatus returned null! >2014-08-05 16:03:35,532 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-82) [3e9bda56] IrsBroker::Failed::GetStoragePoolInfoVDS >2014-08-05 16:03:35,533 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-82) [3e9bda56] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2014-08-05 16:03:35,565 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-15) [2978c3d] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage >2014-08-05 16:03:35,583 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-15) [2978c3d] START, ResetIrsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, ignoreStopFailed = true), log id: 7dfce9c6 >2014-08-05 16:03:35,586 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-15) [2978c3d] START, SpmStopVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9), log id: 653a7005 >2014-08-05 16:03:35,593 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-15) [2978c3d] SpmStopVDSCommand::Stopping SPM on vds rack, pool id 00000002-0002-0002-0002-0000000001d9 >2014-08-05 16:05:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-16) Autorecovering 1 storage domains >2014-08-05 16:05:00,005 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-16) Autorecovering storage domains id: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 >2014-08-05 16:05:00,006 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-16) [4da2a9cc] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage >2014-08-05 16:05:00,008 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-16) [4da2a9cc] ConnectDomainToStorage. Before Connect all hosts to pool. Time:8/5/14 4:05 PM >2014-08-05 16:05:00,016 INFO [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-8-thread-30) [4c845b9c] Running command: ConnectStorageToVdsCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2014-08-05 16:05:00,020 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-30) [4c845b9c] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 1a531241 >2014-08-05 16:05:35,498 WARN [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-15) [2978c3d] could not stop spm of pool 00000002-0002-0002-0002-0000000001d9 on vds 17f7fe56-5f32-40ef-b098-611b84484e0b - reason: org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcRunTimeException: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException >2014-08-05 16:05:35,501 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-15) [2978c3d] FINISH, SpmStopVDSCommand, log id: 653a7005 >2014-08-05 16:05:35,504 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ListVDSCommand] (DefaultQuartzScheduler_Worker-45) [22578e65] Command ListVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-08-05 16:05:35,502 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-30) [4c845b9c] Command ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-08-05 16:05:35,502 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-44) [29bdffa0] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: VDSNetworkException: org.apache.commons.httpclient.NoHttpResponseException: The server 10.34.62.208 failed to respond >2014-08-05 16:05:35,513 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-44) [29bdffa0] hostFromVds::selectedVds - rack, spmStatus returned null! >2014-08-05 16:05:35,511 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-30) [4c845b9c] FINISH, ConnectStorageServerVDSCommand, log id: 1a531241 >2014-08-05 16:05:35,509 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-15) [2978c3d] FINISH, ResetIrsVDSCommand, log id: 7dfce9c6 >2014-08-05 16:05:35,508 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-45) [22578e65] Host rack is not responding. It will stay in Connecting state for a grace period of $80 seconds and after that an attempt to fence the host will be issued. >2014-08-05 16:05:35,516 ERROR [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-8-thread-30) [4c845b9c] Command org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-08-05 16:05:35,518 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-45) [22578e65] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:406) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.fetchRunningVms(VdsUpdateRunTimeInfo.java:971) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVmStats(VdsUpdateRunTimeInfo.java:920) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:638) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:235) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) [:1.7.0_55] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_55] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_55] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_55] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_55] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_55] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_55] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_55] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy82.list(Unknown Source) > at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source) [:1.7.0_55] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_55] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_55] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_55] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_55] > >2014-08-05 16:05:35,529 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-15) [2978c3d] Correlation ID: 2978c3d, Job ID: 4cfc194d-cb73-41b4-aeb7-b2e81a7c24c3, Call Stack: null, Custom Event ID: -1, Message: Failed to Reconstruct Master Domain for Data Center Default. >2014-08-05 16:05:35,524 ERROR [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-8-thread-30) [4c845b9c] Transaction rolled-back for command: org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand. >2014-08-05 16:05:35,551 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-16) [4da2a9cc] ConnectDomainToStorage. After Connect all hosts to pool. Time:8/5/14 4:05 PM >2014-08-05 16:05:35,557 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-46) [3e9bda56] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue >2014-08-05 16:05:35,559 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-35) starting processDomainRecovery for domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 >2014-08-05 16:05:35,565 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-45) [22578e65] Failed to refresh VDS , vds = 17f7fe56-5f32-40ef-b098-611b84484e0b : rack, VDS Network Error, continuing. >java.net.ConnectException: Connection refused >2014-08-05 16:05:35,615 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-35) Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 was reported by all hosts in status UP as problematic. Not moving the domain to NonOperational because it is being reconstructed now. >2014-08-05 16:05:35,634 WARN [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-35) [231778dc] CanDoAction of action ReconstructMasterDomain failed. Reasons:VAR__ACTION__RECONSTRUCT_MASTER,VAR__TYPE__STORAGE__DOMAIN,ACTION_TYPE_FAILED_NO_VDS_IN_POOL >2014-08-05 16:05:35,641 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-35) [231778dc] Removing vds [] from the domain in maintenance cache >2014-08-05 16:05:35,642 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-35) [231778dc] Removing host(s) [] from hosts unseen domain report cache >2014-08-05 16:05:35,643 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-46) [3e9bda56] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue >2014-08-05 16:05:35,645 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-20) starting processDomainRecovery for domain 44559ec0-6bef-491c-bbd6-e67278f37987:export >2014-08-05 16:05:35,650 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-20) Domain 44559ec0-6bef-491c-bbd6-e67278f37987:export was reported by all hosts in status UP as problematic. Moving the domain to NonOperational. >2014-08-05 16:05:35,667 INFO [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-20) [7204577e] Lock Acquired to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE >, sharedLocks= ] >2014-08-05 16:05:35,675 INFO [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-20) [7204577e] Running command: DeactivateStorageDomainCommand internal: true. Entities affected : ID: 44559ec0-6bef-491c-bbd6-e67278f37987 Type: Storage >2014-08-05 16:05:35,682 INFO [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-20) [7204577e] Lock freed to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE >, sharedLocks= ] >2014-08-05 16:05:35,691 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-20) [7204577e] Correlation ID: 7204577e, Job ID: 1424b391-6012-472c-bade-d88fc655c91a, Call Stack: null, Custom Event ID: -1, Message: Storage Domain export (Data Center Default) was deactivated by system because it's not visible by any of the hosts. >2014-08-05 16:05:35,697 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-20) [7204577e] Removing vds [] from the domain in maintenance cache >2014-08-05 16:05:35,698 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-20) [7204577e] Removing host(s) [] from hosts unseen domain report cache >2014-08-05 16:05:38,830 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-25) [32fc3079] START, GetHardwareInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]), log id: d5f83bd >2014-08-05 16:05:38,889 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-25) [32fc3079] FINISH, GetHardwareInfoVDSCommand, log id: d5f83bd >2014-08-05 16:05:38,917 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-25) [5199dc39] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS >2014-08-05 16:05:38,968 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-25) [6de7c8cb] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-08-05 16:05:38,976 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-25) [41f2f1a1] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-08-05 16:05:38,981 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-25) [41f2f1a1] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, storageType = NFS, connectionList = [{ id: 05047c9a-152e-4ef2-b118-4a1753c1eae0, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35export, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 24438db5 >2014-08-05 16:05:39,042 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-25) [41f2f1a1] FINISH, ConnectStorageServerVDSCommand, return: {5083f370-af09-457b-9b73-3440e8b65a07=0, 05047c9a-152e-4ef2-b118-4a1753c1eae0=0}, log id: 24438db5 >2014-08-05 16:05:39,044 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-25) [41f2f1a1] Host rack storage connection was succeeded >2014-08-05 16:05:39,050 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-4) START, ConnectStoragePoolVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, masterVersion = 1), log id: 7208b4b0 >2014-08-05 16:05:45,517 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-26) [6bd9692] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Inactive to Unknown, reason : no reporting hosts >2014-08-05 16:05:45,523 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-26) [6bd9692] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Inactive to Unknown, reason : no reporting hosts >2014-08-05 16:06:39,497 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-4) FINISH, ConnectStoragePoolVDSCommand, log id: 7208b4b0 >2014-08-05 16:06:39,513 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-25) [41f2f1a1] START, SetMOMPolicyParametersVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 3d557516 >2014-08-05 16:06:39,609 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-25) [41f2f1a1] FINISH, SetMOMPolicyParametersVDSCommand, log id: 3d557516 >2014-08-05 16:06:39,617 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-25) [41f2f1a1] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host rack was set to Up. >2014-08-05 16:06:39,627 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-25) [41f2f1a1] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host rack. >2014-08-05 16:06:39,648 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-25) [5937f5d4] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS >2014-08-05 16:06:45,661 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-49) [72c79e4e] hostFromVds::selectedVds - rack, spmStatus Free, storage pool Default >2014-08-05 16:06:45,665 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-49) [72c79e4e] starting spm on vds rack, storage pool Default, prevId -1, LVER -1 >2014-08-05 16:06:45,668 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-49) [72c79e4e] START, SpmStartVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, prevId=-1, prevLVER=-1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 56e233d >2014-08-05 16:06:45,721 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-49) [72c79e4e] spmStart polling started: taskId = 30b57741-adb5-4d27-a2c9-29922b632e26 >2014-08-05 16:06:46,728 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-49) [72c79e4e] spmStart polling ended: taskId = 30b57741-adb5-4d27-a2c9-29922b632e26 task status = finished >2014-08-05 16:06:46,785 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-49) [72c79e4e] spmStart polling ended, spm status: SPM >2014-08-05 16:06:46,788 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-49) [72c79e4e] START, HSMClearTaskVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, taskId=30b57741-adb5-4d27-a2c9-29922b632e26), log id: 26a615b9 >2014-08-05 16:06:46,833 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-49) [72c79e4e] FINISH, HSMClearTaskVDSCommand, log id: 26a615b9 >2014-08-05 16:06:46,834 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-49) [72c79e4e] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@77cb28b8, log id: 56e233d >2014-08-05 16:06:46,838 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-49) [72c79e4e] Initialize Irs proxy from vds: 10.34.62.208 >2014-08-05 16:06:46,846 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-49) [72c79e4e] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208). >2014-08-05 16:06:46,847 WARN [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-8-thread-48) Executing a command: java.util.concurrent.FutureTask , but note that there are 1 tasks in the queue. >2014-08-05 16:06:46,858 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-48) [72c79e4e] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 4c3a9a0c >2014-08-05 16:06:46,885 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-48) [72c79e4e] -- executeIrsBrokerCommand: Attempting on storage pool 00000002-0002-0002-0002-0000000001d9 >2014-08-05 16:06:46,887 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-48) [72c79e4e] START, HSMGetAllTasksInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 10e85e3f >2014-08-05 16:06:46,892 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-48) [72c79e4e] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 10e85e3f >2014-08-05 16:06:46,893 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-48) [72c79e4e] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 4c3a9a0c >2014-08-05 16:06:46,899 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (org.ovirt.thread.pool-8-thread-48) [72c79e4e] Discovered no tasks on Storage Pool Default >2014-08-05 16:23:56,176 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-90) [12692311] Principal admin::internal synchronized >2014-08-05 16:24:25,094 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-37) [10578196] Attempting to update VMs/Templates Ovf. >2014-08-05 17:12:20,470 WARN [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-8-thread-47) Executing a command: java.util.concurrent.FutureTask , but note that there are 1 tasks in the queue. >2014-08-05 17:23:56,182 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-58) [7b19edd8] Principal admin::internal synchronized >2014-08-05 17:24:25,111 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-68) [5d663c2e] Attempting to update VMs/Templates Ovf. >2014-08-05 18:23:56,189 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-40) Principal admin::internal synchronized >2014-08-05 18:24:25,127 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-15) [1ecc4e7f] Attempting to update VMs/Templates Ovf. >2014-08-05 19:06:28,666 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-14) domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 in problem. vds: rack >2014-08-05 19:06:43,788 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-23) domain 44559ec0-6bef-491c-bbd6-e67278f37987:export in problem. vds: rack >2014-08-05 19:08:04,974 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-89) [1b83a377] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2014-08-05 19:08:04,979 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-89) [5784753b] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-08-05 19:08:04,982 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-89) [5784753b] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Active to Unknown, reason : null >2014-08-05 19:08:04,984 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-89) [5784753b] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Active to Unknown, reason : null >2014-08-05 19:08:04,997 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-89) [5784753b] Correlation ID: 5784753b, Call Stack: null, Custom Event ID: -1, Message: Invalid status on Data Center Default. Setting Data Center status to Non Responsive (On host rack, Error: Cannot access master storage domain). >2014-08-05 19:10:05,027 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-89) [5784753b] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2014-08-05 19:10:05,029 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-89) [5784753b] hostFromVds::selectedVds - rack, spmStatus returned null! >2014-08-05 19:10:05,029 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-89) [5784753b] IrsBroker::Failed::GetStoragePoolInfoVDS >2014-08-05 19:10:05,030 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-89) [5784753b] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2014-08-05 19:10:05,048 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-2) [1ba0d6bc] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage >2014-08-05 19:10:05,064 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-2) [1ba0d6bc] START, ResetIrsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, ignoreStopFailed = true), log id: 3ce5e670 >2014-08-05 19:10:05,070 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-2) [1ba0d6bc] START, SpmStopVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9), log id: 2da35dac >2014-08-05 19:10:05,085 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-2) [1ba0d6bc] SpmStopVDSCommand::Stopping SPM on vds rack, pool id 00000002-0002-0002-0002-0000000001d9 >2014-08-05 19:12:04,999 WARN [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-2) [1ba0d6bc] could not stop spm of pool 00000002-0002-0002-0002-0000000001d9 on vds 17f7fe56-5f32-40ef-b098-611b84484e0b - reason: org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcRunTimeException: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException >2014-08-05 19:12:05,001 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-2) [1ba0d6bc] FINISH, SpmStopVDSCommand, log id: 2da35dac >2014-08-05 19:12:05,004 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-93) [71be4699] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: VDSNetworkException: org.apache.commons.httpclient.NoHttpResponseException: The server 10.34.62.208 failed to respond >2014-08-05 19:12:05,003 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetStatsVDSCommand] (DefaultQuartzScheduler_Worker-52) [8557062] Command GetStatsVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-08-05 19:12:05,011 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-2) [1ba0d6bc] FINISH, ResetIrsVDSCommand, log id: 3ce5e670 >2014-08-05 19:12:05,009 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-93) [71be4699] hostFromVds::selectedVds - rack, spmStatus returned null! >2014-08-05 19:12:05,013 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-52) [8557062] Host rack is not responding. It will stay in Connecting state for a grace period of $80 seconds and after that an attempt to fence the host will be issued. >2014-08-05 19:12:05,017 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-52) [8557062] vds::refreshVdsStats Failed getVdsStats, vds = 17f7fe56-5f32-40ef-b098-611b84484e0b : rack, error = VDSNetworkException: java.net.ConnectException: Connection refused >2014-08-05 19:12:05,021 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-52) [8557062] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsStats(VdsUpdateRunTimeInfo.java:669) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:623) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:235) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) [:1.7.0_55] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_55] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_55] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_55] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_55] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_55] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_55] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_55] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy82.getVdsStats(Unknown Source) > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) [:1.7.0_55] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_55] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_55] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_55] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_55] > >2014-08-05 19:12:05,035 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-2) [1ba0d6bc] Correlation ID: 1ba0d6bc, Job ID: b615de3d-7fb9-4a12-a529-e685175fdbd1, Call Stack: null, Custom Event ID: -1, Message: Failed to Reconstruct Master Domain for Data Center Default. >2014-08-05 19:12:05,062 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-27) [5784753b] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue >2014-08-05 19:12:05,064 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-34) starting processDomainRecovery for domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 >2014-08-05 19:12:05,074 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-52) [8557062] Failed to refresh VDS , vds = 17f7fe56-5f32-40ef-b098-611b84484e0b : rack, VDS Network Error, continuing. >java.net.ConnectException: Connection refused >2014-08-05 19:12:05,124 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-34) Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 was reported by all hosts in status UP as problematic. Not moving the domain to NonOperational because it is being reconstructed now. >2014-08-05 19:12:05,141 WARN [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-34) [4ed4667e] CanDoAction of action ReconstructMasterDomain failed. Reasons:VAR__ACTION__RECONSTRUCT_MASTER,VAR__TYPE__STORAGE__DOMAIN,ACTION_TYPE_FAILED_NO_VDS_IN_POOL >2014-08-05 19:12:05,148 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-34) [4ed4667e] Removing vds [] from the domain in maintenance cache >2014-08-05 19:12:05,149 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-34) [4ed4667e] Removing host(s) [] from hosts unseen domain report cache >2014-08-05 19:12:05,150 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-27) [5784753b] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue >2014-08-05 19:12:05,152 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-33) starting processDomainRecovery for domain 44559ec0-6bef-491c-bbd6-e67278f37987:export >2014-08-05 19:12:05,156 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-33) Domain 44559ec0-6bef-491c-bbd6-e67278f37987:export was reported by all hosts in status UP as problematic. Moving the domain to NonOperational. >2014-08-05 19:12:05,171 INFO [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-33) [64ee7315] Lock Acquired to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE >, sharedLocks= ] >2014-08-05 19:12:05,177 INFO [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-33) [64ee7315] Running command: DeactivateStorageDomainCommand internal: true. Entities affected : ID: 44559ec0-6bef-491c-bbd6-e67278f37987 Type: Storage >2014-08-05 19:12:05,182 INFO [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-33) [64ee7315] Lock freed to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE >, sharedLocks= ] >2014-08-05 19:12:05,197 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-33) [64ee7315] Correlation ID: 64ee7315, Job ID: 932e43e0-b76f-4b99-ad80-17ddec28570f, Call Stack: null, Custom Event ID: -1, Message: Storage Domain export (Data Center Default) was deactivated by system because it's not visible by any of the hosts. >2014-08-05 19:12:05,202 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-33) [64ee7315] Removing vds [] from the domain in maintenance cache >2014-08-05 19:12:05,202 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-33) [64ee7315] Removing host(s) [] from hosts unseen domain report cache >2014-08-05 19:12:08,337 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-69) [52d379dc] START, GetHardwareInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]), log id: 560a8536 >2014-08-05 19:12:08,393 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-69) [52d379dc] FINISH, GetHardwareInfoVDSCommand, log id: 560a8536 >2014-08-05 19:12:08,415 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-69) [4ffdb29c] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS >2014-08-05 19:12:08,466 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-69) [b9f6021] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-08-05 19:12:08,473 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-69) [2f905322] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-08-05 19:12:08,485 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-69) [2f905322] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, storageType = NFS, connectionList = [{ id: 05047c9a-152e-4ef2-b118-4a1753c1eae0, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35export, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 226de7da >2014-08-05 19:12:08,546 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-69) [2f905322] FINISH, ConnectStorageServerVDSCommand, return: {5083f370-af09-457b-9b73-3440e8b65a07=0, 05047c9a-152e-4ef2-b118-4a1753c1eae0=0}, log id: 226de7da >2014-08-05 19:12:08,547 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-69) [2f905322] Host rack storage connection was succeeded >2014-08-05 19:12:08,556 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-37) START, ConnectStoragePoolVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, masterVersion = 1), log id: 7319ddb5 >2014-08-05 19:12:15,019 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-40) Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Inactive to Unknown, reason : no reporting hosts >2014-08-05 19:12:15,024 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-40) Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Inactive to Unknown, reason : no reporting hosts >2014-08-05 19:13:08,958 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-37) FINISH, ConnectStoragePoolVDSCommand, log id: 7319ddb5 >2014-08-05 19:13:08,974 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-69) [2f905322] START, SetMOMPolicyParametersVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 4d7c3588 >2014-08-05 19:13:09,066 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-69) [2f905322] FINISH, SetMOMPolicyParametersVDSCommand, log id: 4d7c3588 >2014-08-05 19:13:09,074 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-69) [2f905322] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host rack was set to Up. >2014-08-05 19:13:09,080 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-69) [2f905322] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host rack. >2014-08-05 19:13:09,094 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-69) [5b85dca6] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS >2014-08-05 19:13:15,187 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-65) hostFromVds::selectedVds - rack, spmStatus Free, storage pool Default >2014-08-05 19:13:15,192 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-65) starting spm on vds rack, storage pool Default, prevId -1, LVER -1 >2014-08-05 19:13:15,194 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-65) START, SpmStartVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, prevId=-1, prevLVER=-1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 6b6d70fe >2014-08-05 19:13:15,242 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-65) spmStart polling started: taskId = 600710a5-8f0d-4c9a-8e1b-12d19a74fd62 >2014-08-05 19:13:16,249 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-65) spmStart polling ended: taskId = 600710a5-8f0d-4c9a-8e1b-12d19a74fd62 task status = finished >2014-08-05 19:13:16,304 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-65) spmStart polling ended, spm status: SPM >2014-08-05 19:13:16,307 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-65) START, HSMClearTaskVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, taskId=600710a5-8f0d-4c9a-8e1b-12d19a74fd62), log id: 436b2524 >2014-08-05 19:13:16,352 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-65) FINISH, HSMClearTaskVDSCommand, log id: 436b2524 >2014-08-05 19:13:16,353 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-65) FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@1eb1b0a7, log id: 6b6d70fe >2014-08-05 19:13:16,357 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-65) Initialize Irs proxy from vds: 10.34.62.208 >2014-08-05 19:13:16,364 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-65) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208). >2014-08-05 19:13:16,366 WARN [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-8-thread-4) Executing a command: java.util.concurrent.FutureTask , but note that there are 1 tasks in the queue. >2014-08-05 19:13:16,377 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-4) START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 3239db0f >2014-08-05 19:13:16,405 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-4) -- executeIrsBrokerCommand: Attempting on storage pool 00000002-0002-0002-0002-0000000001d9 >2014-08-05 19:13:16,408 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-4) START, HSMGetAllTasksInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 78209a91 >2014-08-05 19:13:16,413 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-4) FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 78209a91 >2014-08-05 19:13:16,414 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-4) FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 3239db0f >2014-08-05 19:13:16,415 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (org.ovirt.thread.pool-8-thread-4) Discovered no tasks on Storage Pool Default >2014-08-05 19:23:56,194 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-15) [1ecc4e7f] Principal admin::internal synchronized >2014-08-05 19:24:25,145 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-56) [65d9de58] Attempting to update VMs/Templates Ovf. >2014-08-05 20:23:56,200 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-22) [4e09115c] Principal admin::internal synchronized >2014-08-05 20:24:25,159 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-49) [72c79e4e] Attempting to update VMs/Templates Ovf. >2014-08-05 21:23:56,207 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-64) [686e64a1] Principal admin::internal synchronized >2014-08-05 21:24:25,174 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-45) [22578e65] Attempting to update VMs/Templates Ovf. >2014-08-05 22:13:02,270 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-7) domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 in problem. vds: rack >2014-08-05 22:13:02,272 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-7) domain 44559ec0-6bef-491c-bbd6-e67278f37987:export in problem. vds: rack >2014-08-05 22:14:32,081 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-2) Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2014-08-05 22:14:32,085 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-2) [5f3f6fb7] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-08-05 22:14:32,089 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-2) [5f3f6fb7] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Active to Unknown, reason : null >2014-08-05 22:14:32,091 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-2) [5f3f6fb7] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Active to Unknown, reason : null >2014-08-05 22:14:32,107 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-2) [5f3f6fb7] Correlation ID: 5f3f6fb7, Call Stack: null, Custom Event ID: -1, Message: Invalid status on Data Center Default. Setting Data Center status to Non Responsive (On host rack, Error: Cannot access master storage domain). >2014-08-05 22:16:32,131 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-2) [5f3f6fb7] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2014-08-05 22:16:32,133 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-2) [5f3f6fb7] hostFromVds::selectedVds - rack, spmStatus returned null! >2014-08-05 22:16:32,134 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-2) [5f3f6fb7] IrsBroker::Failed::GetStoragePoolInfoVDS >2014-08-05 22:16:32,136 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-2) [5f3f6fb7] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2014-08-05 22:16:32,169 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-23) [2b51aca4] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage >2014-08-05 22:16:32,187 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-23) [2b51aca4] START, ResetIrsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, ignoreStopFailed = true), log id: 5af5ebde >2014-08-05 22:16:32,191 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-23) [2b51aca4] START, SpmStopVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9), log id: 2d1531b1 >2014-08-05 22:16:32,197 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-23) [2b51aca4] SpmStopVDSCommand::Stopping SPM on vds rack, pool id 00000002-0002-0002-0002-0000000001d9 >2014-08-05 22:18:32,102 WARN [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-23) [2b51aca4] could not stop spm of pool 00000002-0002-0002-0002-0000000001d9 on vds 17f7fe56-5f32-40ef-b098-611b84484e0b - reason: org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcRunTimeException: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException >2014-08-05 22:18:32,106 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ListVDSCommand] (DefaultQuartzScheduler_Worker-99) Command ListVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-08-05 22:18:32,107 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-69) [5b85dca6] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: VDSNetworkException: org.apache.commons.httpclient.NoHttpResponseException: The server 10.34.62.208 failed to respond >2014-08-05 22:18:32,111 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-69) [5b85dca6] hostFromVds::selectedVds - rack, spmStatus returned null! >2014-08-05 22:18:32,106 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-23) [2b51aca4] FINISH, SpmStopVDSCommand, log id: 2d1531b1 >2014-08-05 22:18:32,108 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-99) Host rack is not responding. It will stay in Connecting state for a grace period of $80 seconds and after that an attempt to fence the host will be issued. >2014-08-05 22:18:32,114 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-99) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:406) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.fetchRunningVms(VdsUpdateRunTimeInfo.java:971) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVmStats(VdsUpdateRunTimeInfo.java:920) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:638) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:235) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) [:1.7.0_55] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_55] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_55] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_55] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_55] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_55] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_55] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_55] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy82.list(Unknown Source) > at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source) [:1.7.0_55] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_55] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_55] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_55] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_55] > >2014-08-05 22:18:32,118 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-23) [2b51aca4] FINISH, ResetIrsVDSCommand, log id: 5af5ebde >2014-08-05 22:18:32,152 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-99) Failed to refresh VDS , vds = 17f7fe56-5f32-40ef-b098-611b84484e0b : rack, VDS Network Error, continuing. >java.net.ConnectException: Connection refused >2014-08-05 22:18:32,154 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-23) [2b51aca4] Correlation ID: 2b51aca4, Job ID: e287e501-4934-4d16-80c4-74221bf6c77a, Call Stack: null, Custom Event ID: -1, Message: Failed to Reconstruct Master Domain for Data Center Default. >2014-08-05 22:18:32,160 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-25) [5f3f6fb7] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue >2014-08-05 22:18:32,162 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-40) starting processDomainRecovery for domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 >2014-08-05 22:18:32,219 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-40) Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 was reported by all hosts in status UP as problematic. Not moving the domain to NonOperational because it is being reconstructed now. >2014-08-05 22:18:32,236 WARN [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-40) [654db1ff] CanDoAction of action ReconstructMasterDomain failed. Reasons:VAR__ACTION__RECONSTRUCT_MASTER,VAR__TYPE__STORAGE__DOMAIN,ACTION_TYPE_FAILED_NO_VDS_IN_POOL >2014-08-05 22:18:32,242 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-40) [654db1ff] Removing vds [] from the domain in maintenance cache >2014-08-05 22:18:32,243 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-40) [654db1ff] Removing host(s) [] from hosts unseen domain report cache >2014-08-05 22:18:32,244 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-25) [5f3f6fb7] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue >2014-08-05 22:18:32,246 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-4) starting processDomainRecovery for domain 44559ec0-6bef-491c-bbd6-e67278f37987:export >2014-08-05 22:18:32,250 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-4) Domain 44559ec0-6bef-491c-bbd6-e67278f37987:export was reported by all hosts in status UP as problematic. Moving the domain to NonOperational. >2014-08-05 22:18:32,264 INFO [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-4) [3fb49475] Lock Acquired to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE >, sharedLocks= ] >2014-08-05 22:18:32,270 INFO [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-4) [3fb49475] Running command: DeactivateStorageDomainCommand internal: true. Entities affected : ID: 44559ec0-6bef-491c-bbd6-e67278f37987 Type: Storage >2014-08-05 22:18:32,275 INFO [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-4) [3fb49475] Lock freed to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE >, sharedLocks= ] >2014-08-05 22:18:32,284 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-4) [3fb49475] Correlation ID: 3fb49475, Job ID: 9cd5fe91-9ce7-423c-8a88-b7d8729ace8c, Call Stack: null, Custom Event ID: -1, Message: Storage Domain export (Data Center Default) was deactivated by system because it's not visible by any of the hosts. >2014-08-05 22:18:32,289 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-4) [3fb49475] Removing vds [] from the domain in maintenance cache >2014-08-05 22:18:32,290 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-4) [3fb49475] Removing host(s) [] from hosts unseen domain report cache >2014-08-05 22:18:35,229 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-60) [1cb6c42c] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@698df50 >2014-08-05 22:18:35,231 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-60) [1cb6c42c] HostName = rack >2014-08-05 22:18:35,232 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-60) [1cb6c42c] Command GetCapabilitiesVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]) execution failed. Exception: VDSRecoveringException: Recovering from crash or Initializing >2014-08-05 22:18:35,247 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-60) [1cb6c42c] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSRecoveringException: Recovering from crash or Initializing > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:16) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:566) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:235) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Host rack is initializing. Message: Recovering from crash or Initializing >2014-08-05 22:18:35,254 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-60) [1cb6c42c] Failed to refresh VDS , vds = 17f7fe56-5f32-40ef-b098-611b84484e0b : rack, error = Recovering from crash or Initializing, continuing. >2014-08-05 22:18:38,484 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-61) [58bfe349] START, GetHardwareInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]), log id: 520c9d01 >2014-08-05 22:18:38,535 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-61) [58bfe349] FINISH, GetHardwareInfoVDSCommand, log id: 520c9d01 >2014-08-05 22:18:38,556 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-61) [155dec1] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS >2014-08-05 22:18:38,566 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-61) [155dec1] Cancelling the recovery from crash timer for VDS 17f7fe56-5f32-40ef-b098-611b84484e0b because vds started initializing >2014-08-05 22:18:38,568 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-61) [3d09986d] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-08-05 22:18:38,575 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-61) [424859a7] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-08-05 22:18:38,580 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-61) [424859a7] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, storageType = NFS, connectionList = [{ id: 05047c9a-152e-4ef2-b118-4a1753c1eae0, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35export, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 1b8c12a1 >2014-08-05 22:18:38,641 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-61) [424859a7] FINISH, ConnectStorageServerVDSCommand, return: {5083f370-af09-457b-9b73-3440e8b65a07=0, 05047c9a-152e-4ef2-b118-4a1753c1eae0=0}, log id: 1b8c12a1 >2014-08-05 22:18:38,642 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-61) [424859a7] Host rack storage connection was succeeded >2014-08-05 22:18:38,648 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-14) START, ConnectStoragePoolVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, masterVersion = 1), log id: 585e3da2 >2014-08-05 22:18:42,115 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-30) [3adcf236] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Inactive to Unknown, reason : no reporting hosts >2014-08-05 22:18:42,120 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-30) [3adcf236] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Inactive to Unknown, reason : no reporting hosts >2014-08-05 22:19:39,055 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-14) FINISH, ConnectStoragePoolVDSCommand, log id: 585e3da2 >2014-08-05 22:19:39,068 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-61) [424859a7] START, SetMOMPolicyParametersVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 687d7a19 >2014-08-05 22:19:39,161 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-61) [424859a7] FINISH, SetMOMPolicyParametersVDSCommand, log id: 687d7a19 >2014-08-05 22:19:39,169 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-61) [424859a7] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host rack was set to Up. >2014-08-05 22:19:39,176 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-61) [424859a7] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host rack. >2014-08-05 22:19:39,205 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-61) [240369c1] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS >2014-08-05 22:19:42,279 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-12) [5b5ccf8] hostFromVds::selectedVds - rack, spmStatus Free, storage pool Default >2014-08-05 22:19:42,284 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-12) [5b5ccf8] starting spm on vds rack, storage pool Default, prevId -1, LVER -1 >2014-08-05 22:19:42,287 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-12) [5b5ccf8] START, SpmStartVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, prevId=-1, prevLVER=-1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 56424d6d >2014-08-05 22:19:42,334 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-12) [5b5ccf8] spmStart polling started: taskId = fbaaa79c-16f9-4470-8dc9-f22761822adb >2014-08-05 22:19:43,341 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-12) [5b5ccf8] spmStart polling ended: taskId = fbaaa79c-16f9-4470-8dc9-f22761822adb task status = finished >2014-08-05 22:19:43,398 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-12) [5b5ccf8] spmStart polling ended, spm status: SPM >2014-08-05 22:19:43,400 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-12) [5b5ccf8] START, HSMClearTaskVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, taskId=fbaaa79c-16f9-4470-8dc9-f22761822adb), log id: 19951f7 >2014-08-05 22:19:43,445 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-12) [5b5ccf8] FINISH, HSMClearTaskVDSCommand, log id: 19951f7 >2014-08-05 22:19:43,446 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-12) [5b5ccf8] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@5d5dc29e, log id: 56424d6d >2014-08-05 22:19:43,451 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-12) [5b5ccf8] Initialize Irs proxy from vds: 10.34.62.208 >2014-08-05 22:19:43,457 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-12) [5b5ccf8] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208). >2014-08-05 22:19:43,458 WARN [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-8-thread-15) Executing a command: java.util.concurrent.FutureTask , but note that there are 1 tasks in the queue. >2014-08-05 22:19:43,471 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-15) [5b5ccf8] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 2e7036e7 >2014-08-05 22:19:43,496 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-15) [5b5ccf8] -- executeIrsBrokerCommand: Attempting on storage pool 00000002-0002-0002-0002-0000000001d9 >2014-08-05 22:19:43,498 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-15) [5b5ccf8] START, HSMGetAllTasksInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 6bb68551 >2014-08-05 22:19:43,503 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-15) [5b5ccf8] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 6bb68551 >2014-08-05 22:19:43,504 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-15) [5b5ccf8] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 2e7036e7 >2014-08-05 22:19:43,505 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (org.ovirt.thread.pool-8-thread-15) [5b5ccf8] Discovered no tasks on Storage Pool Default >2014-08-05 22:23:56,213 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-44) [29bdffa0] Principal admin::internal synchronized >2014-08-05 22:24:25,190 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-28) [8ef1638] Attempting to update VMs/Templates Ovf. >2014-08-05 23:23:56,218 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-52) [8557062] Principal admin::internal synchronized >2014-08-05 23:24:25,220 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-8) [782c75f] Attempting to update VMs/Templates Ovf. >2014-08-06 00:23:56,227 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-63) Principal admin::internal synchronized >2014-08-06 00:24:25,235 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-40) Attempting to update VMs/Templates Ovf. >2014-08-06 01:19:35,727 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-48) domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 in problem. vds: rack >2014-08-06 01:19:35,729 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-48) domain 44559ec0-6bef-491c-bbd6-e67278f37987:export in problem. vds: rack >2014-08-06 01:21:01,726 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-21) [675f1edc] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2014-08-06 01:21:01,731 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-21) [6e64f208] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-08-06 01:21:01,735 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-21) [6e64f208] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Active to Unknown, reason : null >2014-08-06 01:21:01,737 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-21) [6e64f208] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Active to Unknown, reason : null >2014-08-06 01:21:01,748 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-21) [6e64f208] Correlation ID: 6e64f208, Call Stack: null, Custom Event ID: -1, Message: Invalid status on Data Center Default. Setting Data Center status to Non Responsive (On host rack, Error: Cannot access master storage domain). >2014-08-06 01:23:01,783 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-21) [6e64f208] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2014-08-06 01:23:01,785 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-21) [6e64f208] hostFromVds::selectedVds - rack, spmStatus returned null! >2014-08-06 01:23:01,786 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-21) [6e64f208] IrsBroker::Failed::GetStoragePoolInfoVDS >2014-08-06 01:23:01,787 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-21) [6e64f208] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2014-08-06 01:23:01,821 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-9) [6fc5f0a] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage >2014-08-06 01:23:01,839 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-9) [6fc5f0a] START, ResetIrsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, ignoreStopFailed = true), log id: 124023c5 >2014-08-06 01:23:01,843 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-9) [6fc5f0a] START, SpmStopVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9), log id: 74dafaf5 >2014-08-06 01:23:01,857 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-9) [6fc5f0a] SpmStopVDSCommand::Stopping SPM on vds rack, pool id 00000002-0002-0002-0002-0000000001d9 >2014-08-06 01:23:56,233 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-89) [5784753b] Principal admin::internal synchronized >2014-08-06 01:24:25,251 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-57) [cd4c3ce] Attempting to update VMs/Templates Ovf. >2014-08-06 01:25:00,003 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-15) [1ecc4e7f] Autorecovering 1 storage domains >2014-08-06 01:25:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-15) [1ecc4e7f] Autorecovering storage domains id: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 >2014-08-06 01:25:00,006 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-15) [7e39aaf1] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage >2014-08-06 01:25:00,008 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-15) [7e39aaf1] ConnectDomainToStorage. Before Connect all hosts to pool. Time:8/6/14 1:25 AM >2014-08-06 01:25:00,017 INFO [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-8-thread-16) [61fef78c] Running command: ConnectStorageToVdsCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2014-08-06 01:25:00,021 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-16) [61fef78c] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 4c704cb3 >2014-08-06 01:25:01,773 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-34) [36ef1be6] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2014-08-06 01:25:01,775 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-34) [36ef1be6] hostFromVds::selectedVds - rack, spmStatus returned null! >2014-08-06 01:25:01,776 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-34) [36ef1be6] IrsBroker::Failed::GetStoragePoolInfoVDS >2014-08-06 01:25:01,777 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-34) [36ef1be6] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2014-08-06 01:26:01,859 WARN [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-9) [6fc5f0a] could not stop spm of pool 00000002-0002-0002-0002-0000000001d9 on vds 17f7fe56-5f32-40ef-b098-611b84484e0b - reason: org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcRunTimeException: java.util.concurrent.TimeoutException >2014-08-06 01:26:01,861 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-9) [6fc5f0a] FINISH, SpmStopVDSCommand, log id: 74dafaf5 >2014-08-06 01:26:01,868 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-9) [6fc5f0a] FINISH, ResetIrsVDSCommand, log id: 124023c5 >2014-08-06 01:26:01,877 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-16) [61fef78c] FINISH, ConnectStorageServerVDSCommand, return: {5083f370-af09-457b-9b73-3440e8b65a07=0}, log id: 4c704cb3 >2014-08-06 01:26:01,881 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-15) [7e39aaf1] ConnectDomainToStorage. After Connect all hosts to pool. Time:8/6/14 1:26 AM >2014-08-06 01:26:01,887 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-9) [6fc5f0a] Correlation ID: 6fc5f0a, Job ID: a0b981be-04f8-4b11-b08e-7585a91456da, Call Stack: null, Custom Event ID: -1, Message: Failed to Reconstruct Master Domain for Data Center Default. >2014-08-06 01:26:01,893 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-33) [6e64f208] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue >2014-08-06 01:26:01,896 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-48) starting processDomainRecovery for domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 >2014-08-06 01:26:01,959 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-48) Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 was reported by all hosts in status UP as problematic. Not moving the domain to NonOperational because it is being reconstructed now. >2014-08-06 01:26:01,984 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-48) [12ddcc28] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage >2014-08-06 01:26:02,011 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-48) [12ddcc28] Removing vds [] from the domain in maintenance cache >2014-08-06 01:26:02,011 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-48) [12ddcc28] Removing host(s) [] from hosts unseen domain report cache >2014-08-06 01:26:02,012 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-33) [6e64f208] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue >2014-08-06 01:26:02,015 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-10) starting processDomainRecovery for domain 44559ec0-6bef-491c-bbd6-e67278f37987:export >2014-08-06 01:26:02,020 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-10) Domain 44559ec0-6bef-491c-bbd6-e67278f37987:export was reported by all hosts in status UP as problematic. Moving the domain to NonOperational. >2014-08-06 01:26:02,037 INFO [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-10) [215eea0] Lock Acquired to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE >, sharedLocks= ] >2014-08-06 01:26:02,043 INFO [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-10) [215eea0] Running command: DeactivateStorageDomainCommand internal: true. Entities affected : ID: 44559ec0-6bef-491c-bbd6-e67278f37987 Type: Storage >2014-08-06 01:26:02,049 INFO [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-10) [215eea0] Lock freed to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE >, sharedLocks= ] >2014-08-06 01:26:02,058 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-10) [215eea0] Correlation ID: 215eea0, Job ID: c90ed536-d1b3-4e2b-b1a5-aa5980d65bcd, Call Stack: null, Custom Event ID: -1, Message: Storage Domain export (Data Center Default) was deactivated by system because it's not visible by any of the hosts. >2014-08-06 01:26:02,063 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-10) [215eea0] Removing vds [] from the domain in maintenance cache >2014-08-06 01:26:02,064 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-10) [215eea0] Removing host(s) [] from hosts unseen domain report cache >2014-08-06 01:27:01,784 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-10) Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: VDSNetworkException: org.apache.commons.httpclient.NoHttpResponseException: The server 10.34.62.208 failed to respond >2014-08-06 01:27:01,787 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-10) hostFromVds::selectedVds - rack, spmStatus returned null! >2014-08-06 01:27:02,510 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ListVDSCommand] (DefaultQuartzScheduler_Worker-68) [5d663c2e] Command ListVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]) execution failed. Exception: VDSRecoveringException: Recovering from crash or Initializing >2014-08-06 01:27:02,511 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-68) [5d663c2e] Failed vds listing, vds = 17f7fe56-5f32-40ef-b098-611b84484e0b : rack, error = org.ovirt.engine.core.vdsbroker.vdsbroker.VDSRecoveringException: Recovering from crash or Initializing >2014-08-06 01:27:02,527 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-68) [5d663c2e] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSRecoveringException: Recovering from crash or Initializing > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.vdsbroker.ListVDSCommand.executeVdsBrokerCommand(ListVDSCommand.java:24) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:406) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.fetchRunningVms(VdsUpdateRunTimeInfo.java:971) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVmStats(VdsUpdateRunTimeInfo.java:920) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:638) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:235) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Host rack is initializing. Message: Recovering from crash or Initializing >2014-08-06 01:27:02,536 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-68) [5d663c2e] Failed to refresh VDS , vds = 17f7fe56-5f32-40ef-b098-611b84484e0b : rack, error = Recovering from crash or Initializing, continuing. >2014-08-06 01:27:05,772 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-45) [22578e65] START, GetHardwareInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]), log id: 2ac8372a >2014-08-06 01:27:05,824 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-45) [22578e65] FINISH, GetHardwareInfoVDSCommand, log id: 2ac8372a >2014-08-06 01:27:05,851 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-45) [349ba706] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS >2014-08-06 01:27:05,900 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-45) [349ba706] Cancelling the recovery from crash timer for VDS 17f7fe56-5f32-40ef-b098-611b84484e0b because vds started initializing >2014-08-06 01:27:05,902 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-45) [60eb2609] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-08-06 01:27:05,910 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-45) [6513a350] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-08-06 01:27:05,922 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-45) [6513a350] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, storageType = NFS, connectionList = [{ id: 05047c9a-152e-4ef2-b118-4a1753c1eae0, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35export, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 46cc6880 >2014-08-06 01:27:05,987 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-45) [6513a350] FINISH, ConnectStorageServerVDSCommand, return: {5083f370-af09-457b-9b73-3440e8b65a07=0, 05047c9a-152e-4ef2-b118-4a1753c1eae0=0}, log id: 46cc6880 >2014-08-06 01:27:05,994 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-45) [6513a350] Host rack storage connection was succeeded >2014-08-06 01:27:06,009 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-35) START, ConnectStoragePoolVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, masterVersion = 1), log id: 6124fc25 >2014-08-06 01:27:11,792 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-63) Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Inactive to Unknown, reason : no reporting hosts >2014-08-06 01:27:11,797 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-63) Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Inactive to Unknown, reason : no reporting hosts >2014-08-06 01:28:06,424 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-35) FINISH, ConnectStoragePoolVDSCommand, log id: 6124fc25 >2014-08-06 01:28:06,438 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-45) [6513a350] START, SetMOMPolicyParametersVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 1d96cbbf >2014-08-06 01:28:06,532 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-45) [6513a350] FINISH, SetMOMPolicyParametersVDSCommand, log id: 1d96cbbf >2014-08-06 01:28:06,539 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-45) [6513a350] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host rack was set to Up. >2014-08-06 01:28:06,545 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-45) [6513a350] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host rack. >2014-08-06 01:28:06,558 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-45) [6a31c23d] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS >2014-08-06 01:28:11,910 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-54) [23c334ab] hostFromVds::selectedVds - rack, spmStatus Free, storage pool Default >2014-08-06 01:28:11,916 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-54) [23c334ab] starting spm on vds rack, storage pool Default, prevId -1, LVER -1 >2014-08-06 01:28:11,918 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-54) [23c334ab] START, SpmStartVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, prevId=-1, prevLVER=-1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 2123d38a >2014-08-06 01:28:11,965 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-54) [23c334ab] spmStart polling started: taskId = b07bd89b-5712-4747-bd9c-2c8684c1beb3 >2014-08-06 01:28:12,972 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-54) [23c334ab] spmStart polling ended: taskId = b07bd89b-5712-4747-bd9c-2c8684c1beb3 task status = finished >2014-08-06 01:28:13,027 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-54) [23c334ab] spmStart polling ended, spm status: SPM >2014-08-06 01:28:13,029 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-54) [23c334ab] START, HSMClearTaskVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, taskId=b07bd89b-5712-4747-bd9c-2c8684c1beb3), log id: 581b2fd9 >2014-08-06 01:28:13,074 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-54) [23c334ab] FINISH, HSMClearTaskVDSCommand, log id: 581b2fd9 >2014-08-06 01:28:13,075 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-54) [23c334ab] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@6189f619, log id: 2123d38a >2014-08-06 01:28:13,079 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-54) [23c334ab] Initialize Irs proxy from vds: 10.34.62.208 >2014-08-06 01:28:13,085 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-54) [23c334ab] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208). >2014-08-06 01:28:13,095 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-37) [23c334ab] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 32a00994 >2014-08-06 01:28:13,129 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-37) [23c334ab] -- executeIrsBrokerCommand: Attempting on storage pool 00000002-0002-0002-0002-0000000001d9 >2014-08-06 01:28:13,131 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-37) [23c334ab] START, HSMGetAllTasksInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 32b89fb4 >2014-08-06 01:28:13,136 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-37) [23c334ab] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 32b89fb4 >2014-08-06 01:28:13,137 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-37) [23c334ab] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 32a00994 >2014-08-06 01:28:13,138 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (org.ovirt.thread.pool-8-thread-37) [23c334ab] Discovered no tasks on Storage Pool Default >2014-08-06 02:23:56,239 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-29) [327b509d] Principal admin::internal synchronized >2014-08-06 02:24:25,253 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-38) [4bd2cb12] Attempting to update VMs/Templates Ovf. >2014-08-06 03:23:56,244 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-77) [1f7efd53] Principal admin::internal synchronized >2014-08-06 03:24:25,267 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-8) [782c75f] Attempting to update VMs/Templates Ovf. >2014-08-06 03:35:35,001 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-12) [5b5ccf8] Start deleteAgedOutAuditLogs >2014-08-06 03:35:35,002 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-73) [55405a19] Start deleteAgedOutCommandEntities >2014-08-06 03:35:35,010 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-73) [55405a19] Finished deleteAgedOutCommandEntities >2014-08-06 03:35:35,016 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-12) [5b5ccf8] Finished deleteAgedOutAuditLogs >2014-08-06 04:23:56,249 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-98) [5aa74089] Principal admin::internal synchronized >2014-08-06 04:24:25,283 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-59) [1f7257b7] Attempting to update VMs/Templates Ovf. >2014-08-06 04:27:50,606 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-16) domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 in problem. vds: rack >2014-08-06 04:27:50,608 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-16) domain 44559ec0-6bef-491c-bbd6-e67278f37987:export in problem. vds: rack >2014-08-06 04:29:29,818 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-99) Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2014-08-06 04:29:29,823 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-99) [39ee7ea8] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-08-06 04:29:29,828 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-99) [39ee7ea8] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Active to Unknown, reason : null >2014-08-06 04:29:29,832 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-99) [39ee7ea8] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Active to Unknown, reason : null >2014-08-06 04:29:29,849 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-99) [39ee7ea8] Correlation ID: 39ee7ea8, Call Stack: null, Custom Event ID: -1, Message: Invalid status on Data Center Default. Setting Data Center status to Non Responsive (On host rack, Error: Cannot access master storage domain). >2014-08-06 04:31:29,869 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-99) [39ee7ea8] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2014-08-06 04:31:29,871 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-99) [39ee7ea8] hostFromVds::selectedVds - rack, spmStatus returned null! >2014-08-06 04:31:29,872 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-99) [39ee7ea8] IrsBroker::Failed::GetStoragePoolInfoVDS >2014-08-06 04:31:29,873 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-99) [39ee7ea8] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2014-08-06 04:31:29,918 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-15) [24d41939] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage >2014-08-06 04:31:29,935 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-15) [24d41939] START, ResetIrsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, ignoreStopFailed = true), log id: 2fe73741 >2014-08-06 04:31:29,938 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-15) [24d41939] START, SpmStopVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9), log id: 1373a335 >2014-08-06 04:31:29,944 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-15) [24d41939] SpmStopVDSCommand::Stopping SPM on vds rack, pool id 00000002-0002-0002-0002-0000000001d9 >2014-08-06 04:33:29,842 WARN [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-15) [24d41939] could not stop spm of pool 00000002-0002-0002-0002-0000000001d9 on vds 17f7fe56-5f32-40ef-b098-611b84484e0b - reason: org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcRunTimeException: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException >2014-08-06 04:33:29,845 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-15) [24d41939] FINISH, SpmStopVDSCommand, log id: 1373a335 >2014-08-06 04:33:29,846 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ListVDSCommand] (DefaultQuartzScheduler_Worker-90) [12692311] Command ListVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-08-06 04:33:29,847 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-79) [37510499] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: VDSNetworkException: org.apache.commons.httpclient.NoHttpResponseException: The server 10.34.62.208 failed to respond >2014-08-06 04:33:29,848 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-90) [12692311] Host rack is not responding. It will stay in Connecting state for a grace period of $80 seconds and after that an attempt to fence the host will be issued. >2014-08-06 04:33:29,851 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-79) [37510499] hostFromVds::selectedVds - rack, spmStatus returned null! >2014-08-06 04:33:29,852 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-90) [12692311] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:406) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.fetchRunningVms(VdsUpdateRunTimeInfo.java:971) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVmStats(VdsUpdateRunTimeInfo.java:920) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:638) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:235) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) [:1.7.0_55] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_55] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_55] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_55] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_55] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_55] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_55] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_55] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy82.list(Unknown Source) > at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source) [:1.7.0_55] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_55] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_55] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_55] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_55] > >2014-08-06 04:33:29,856 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-15) [24d41939] FINISH, ResetIrsVDSCommand, log id: 2fe73741 >2014-08-06 04:33:29,885 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-90) [12692311] Failed to refresh VDS , vds = 17f7fe56-5f32-40ef-b098-611b84484e0b : rack, VDS Network Error, continuing. >java.net.ConnectException: Connection refused >2014-08-06 04:33:29,891 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-15) [24d41939] Correlation ID: 24d41939, Job ID: 831f1f82-0fdf-458a-bbfe-b00c578ead72, Call Stack: null, Custom Event ID: -1, Message: Failed to Reconstruct Master Domain for Data Center Default. >2014-08-06 04:33:29,897 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-42) [39ee7ea8] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue >2014-08-06 04:33:29,899 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-45) starting processDomainRecovery for domain 44559ec0-6bef-491c-bbd6-e67278f37987:export >2014-08-06 04:33:29,953 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-45) Domain 44559ec0-6bef-491c-bbd6-e67278f37987:export was reported by all hosts in status UP as problematic. Moving the domain to NonOperational. >2014-08-06 04:33:29,969 INFO [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-45) [2f86b7e4] Lock Acquired to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE >, sharedLocks= ] >2014-08-06 04:33:29,975 INFO [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-45) [2f86b7e4] Running command: DeactivateStorageDomainCommand internal: true. Entities affected : ID: 44559ec0-6bef-491c-bbd6-e67278f37987 Type: Storage >2014-08-06 04:33:29,980 INFO [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-45) [2f86b7e4] Lock freed to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE >, sharedLocks= ] >2014-08-06 04:33:29,989 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-45) [2f86b7e4] Correlation ID: 2f86b7e4, Job ID: 08f731d2-80ec-4b96-9c96-306a760a4282, Call Stack: null, Custom Event ID: -1, Message: Storage Domain export (Data Center Default) was deactivated by system because it's not visible by any of the hosts. >2014-08-06 04:33:29,994 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-45) [2f86b7e4] Removing vds [] from the domain in maintenance cache >2014-08-06 04:33:29,995 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-45) [2f86b7e4] Removing host(s) [] from hosts unseen domain report cache >2014-08-06 04:33:29,997 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-48) starting processDomainRecovery for domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 >2014-08-06 04:33:30,001 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-48) Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 was reported by all hosts in status UP as problematic. Not moving the domain to NonOperational because it is being reconstructed now. >2014-08-06 04:33:30,019 WARN [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-48) [7a434c52] CanDoAction of action ReconstructMasterDomain failed. Reasons:VAR__ACTION__RECONSTRUCT_MASTER,VAR__TYPE__STORAGE__DOMAIN,ACTION_TYPE_FAILED_NO_VDS_IN_POOL >2014-08-06 04:33:30,025 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-48) [7a434c52] Removing vds [] from the domain in maintenance cache >2014-08-06 04:33:30,026 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-48) [7a434c52] Removing host(s) [] from hosts unseen domain report cache >2014-08-06 04:33:30,027 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-42) [39ee7ea8] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue >2014-08-06 04:33:33,145 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-20) [367a2664] START, GetHardwareInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]), log id: 23d7d633 >2014-08-06 04:33:33,194 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-20) [367a2664] FINISH, GetHardwareInfoVDSCommand, log id: 23d7d633 >2014-08-06 04:33:33,218 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-20) [7e4cbe3c] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS >2014-08-06 04:33:33,268 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-20) [16648d26] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-08-06 04:33:33,275 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-20) [665369f8] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-08-06 04:33:33,281 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-20) [665369f8] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, storageType = NFS, connectionList = [{ id: 05047c9a-152e-4ef2-b118-4a1753c1eae0, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35export, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 7ded8a61 >2014-08-06 04:33:33,340 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-20) [665369f8] FINISH, ConnectStorageServerVDSCommand, return: {5083f370-af09-457b-9b73-3440e8b65a07=0, 05047c9a-152e-4ef2-b118-4a1753c1eae0=0}, log id: 7ded8a61 >2014-08-06 04:33:33,342 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-20) [665369f8] Host rack storage connection was succeeded >2014-08-06 04:33:33,348 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-37) START, ConnectStoragePoolVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, masterVersion = 1), log id: 6afd7327 >2014-08-06 04:33:39,858 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-4) Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Inactive to Unknown, reason : no reporting hosts >2014-08-06 04:33:39,862 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-4) Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Inactive to Unknown, reason : no reporting hosts >2014-08-06 04:34:33,758 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-37) FINISH, ConnectStoragePoolVDSCommand, log id: 6afd7327 >2014-08-06 04:34:33,771 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-20) [665369f8] START, SetMOMPolicyParametersVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 16c582a7 >2014-08-06 04:34:33,865 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-20) [665369f8] FINISH, SetMOMPolicyParametersVDSCommand, log id: 16c582a7 >2014-08-06 04:34:33,872 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-20) [665369f8] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host rack was set to Up. >2014-08-06 04:34:33,878 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-20) [665369f8] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host rack. >2014-08-06 04:34:33,894 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-20) [6d05f75f] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS >2014-08-06 04:34:39,978 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-47) hostFromVds::selectedVds - rack, spmStatus Free, storage pool Default >2014-08-06 04:34:39,984 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-47) starting spm on vds rack, storage pool Default, prevId -1, LVER -1 >2014-08-06 04:34:39,985 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-47) START, SpmStartVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, prevId=-1, prevLVER=-1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 1f58b41a >2014-08-06 04:34:40,034 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-47) spmStart polling started: taskId = 79a8065d-89ce-4d94-9aa3-0363ad72dca5 >2014-08-06 04:34:41,040 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-47) spmStart polling ended: taskId = 79a8065d-89ce-4d94-9aa3-0363ad72dca5 task status = finished >2014-08-06 04:34:41,096 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-47) spmStart polling ended, spm status: SPM >2014-08-06 04:34:41,099 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-47) START, HSMClearTaskVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, taskId=79a8065d-89ce-4d94-9aa3-0363ad72dca5), log id: 1e23ed72 >2014-08-06 04:34:41,143 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-47) FINISH, HSMClearTaskVDSCommand, log id: 1e23ed72 >2014-08-06 04:34:41,144 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-47) FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@588896ea, log id: 1f58b41a >2014-08-06 04:34:41,147 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-47) Initialize Irs proxy from vds: 10.34.62.208 >2014-08-06 04:34:41,153 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-47) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208). >2014-08-06 04:34:41,162 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-33) START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 3b5dce45 >2014-08-06 04:34:41,192 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-33) -- executeIrsBrokerCommand: Attempting on storage pool 00000002-0002-0002-0002-0000000001d9 >2014-08-06 04:34:41,194 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-33) START, HSMGetAllTasksInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 50d7369b >2014-08-06 04:34:41,200 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-33) FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 50d7369b >2014-08-06 04:34:41,201 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-33) FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 3b5dce45 >2014-08-06 04:34:41,202 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (org.ovirt.thread.pool-8-thread-33) Discovered no tasks on Storage Pool Default >2014-08-06 05:23:56,257 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-43) [56fe64d8] Principal admin::internal synchronized >2014-08-06 05:24:25,297 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-1) [795e8140] Attempting to update VMs/Templates Ovf. >2014-08-06 06:23:56,264 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-56) [65d9de58] Principal admin::internal synchronized >2014-08-06 06:24:25,331 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-67) [6199900c] Attempting to update VMs/Templates Ovf. >2014-08-06 07:23:56,271 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-59) [1f7257b7] Principal admin::internal synchronized >2014-08-06 07:24:25,345 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-26) [6bd9692] Attempting to update VMs/Templates Ovf. >2014-08-06 07:34:25,187 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-29) domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 in problem. vds: rack >2014-08-06 07:34:25,190 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-29) domain 44559ec0-6bef-491c-bbd6-e67278f37987:export in problem. vds: rack >2014-08-06 07:35:56,497 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-54) [23c334ab] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2014-08-06 07:35:56,501 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-54) [75c8e759] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-08-06 07:35:56,505 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-54) [75c8e759] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Active to Unknown, reason : null >2014-08-06 07:35:56,507 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-54) [75c8e759] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Active to Unknown, reason : null >2014-08-06 07:35:56,531 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-54) [75c8e759] Correlation ID: 75c8e759, Call Stack: null, Custom Event ID: -1, Message: Invalid status on Data Center Default. Setting Data Center status to Non Responsive (On host rack, Error: Cannot access master storage domain). >2014-08-06 07:37:56,549 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-54) [75c8e759] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2014-08-06 07:37:56,552 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-54) [75c8e759] hostFromVds::selectedVds - rack, spmStatus returned null! >2014-08-06 07:37:56,553 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-54) [75c8e759] IrsBroker::Failed::GetStoragePoolInfoVDS >2014-08-06 07:37:56,554 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-54) [75c8e759] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2014-08-06 07:37:56,587 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-47) [38bb2b88] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage >2014-08-06 07:37:56,606 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-47) [38bb2b88] START, ResetIrsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, ignoreStopFailed = true), log id: 1d0db938 >2014-08-06 07:37:56,609 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-47) [38bb2b88] START, SpmStopVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9), log id: 1ada5348 >2014-08-06 07:37:56,615 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-47) [38bb2b88] SpmStopVDSCommand::Stopping SPM on vds rack, pool id 00000002-0002-0002-0002-0000000001d9 >2014-08-06 07:39:56,519 WARN [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-47) [38bb2b88] could not stop spm of pool 00000002-0002-0002-0002-0000000001d9 on vds 17f7fe56-5f32-40ef-b098-611b84484e0b - reason: org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcRunTimeException: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException >2014-08-06 07:39:56,524 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-47) [38bb2b88] FINISH, SpmStopVDSCommand, log id: 1ada5348 >2014-08-06 07:39:56,524 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-50) [16080eb6] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: VDSNetworkException: org.apache.commons.httpclient.NoHttpResponseException: The server 10.34.62.208 failed to respond >2014-08-06 07:39:56,525 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ListVDSCommand] (DefaultQuartzScheduler_Worker-94) Command ListVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-08-06 07:39:56,529 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-50) [16080eb6] hostFromVds::selectedVds - rack, spmStatus returned null! >2014-08-06 07:39:56,529 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-94) Host rack is not responding. It will stay in Connecting state for a grace period of $80 seconds and after that an attempt to fence the host will be issued. >2014-08-06 07:39:56,532 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-47) [38bb2b88] FINISH, ResetIrsVDSCommand, log id: 1d0db938 >2014-08-06 07:39:56,531 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-94) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:406) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.fetchRunningVms(VdsUpdateRunTimeInfo.java:971) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVmStats(VdsUpdateRunTimeInfo.java:920) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:638) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:235) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) [:1.7.0_55] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_55] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_55] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_55] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_55] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_55] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_55] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_55] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy82.list(Unknown Source) > at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source) [:1.7.0_55] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_55] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_55] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_55] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_55] > >2014-08-06 07:39:56,541 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-47) [38bb2b88] Correlation ID: 38bb2b88, Job ID: 9a102b86-0aa0-456d-9c0d-e517af511753, Call Stack: null, Custom Event ID: -1, Message: Failed to Reconstruct Master Domain for Data Center Default. >2014-08-06 07:39:56,561 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-94) Failed to refresh VDS , vds = 17f7fe56-5f32-40ef-b098-611b84484e0b : rack, VDS Network Error, continuing. >java.net.ConnectException: Connection refused >2014-08-06 07:39:56,565 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-44) [75c8e759] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue >2014-08-06 07:39:56,566 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-29) starting processDomainRecovery for domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 >2014-08-06 07:39:56,630 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-29) Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 was reported by all hosts in status UP as problematic. Not moving the domain to NonOperational because it is being reconstructed now. >2014-08-06 07:39:56,654 WARN [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-29) [1aa8c125] CanDoAction of action ReconstructMasterDomain failed. Reasons:VAR__ACTION__RECONSTRUCT_MASTER,VAR__TYPE__STORAGE__DOMAIN,ACTION_TYPE_FAILED_NO_VDS_IN_POOL >2014-08-06 07:39:56,661 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-29) [1aa8c125] Removing vds [] from the domain in maintenance cache >2014-08-06 07:39:56,662 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-29) [1aa8c125] Removing host(s) [] from hosts unseen domain report cache >2014-08-06 07:39:56,663 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-44) [75c8e759] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue >2014-08-06 07:39:56,664 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-48) starting processDomainRecovery for domain 44559ec0-6bef-491c-bbd6-e67278f37987:export >2014-08-06 07:39:56,669 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-48) Domain 44559ec0-6bef-491c-bbd6-e67278f37987:export was reported by all hosts in status UP as problematic. Moving the domain to NonOperational. >2014-08-06 07:39:56,684 INFO [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-48) [349f4a8e] Lock Acquired to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE >, sharedLocks= ] >2014-08-06 07:39:56,691 INFO [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-48) [349f4a8e] Running command: DeactivateStorageDomainCommand internal: true. Entities affected : ID: 44559ec0-6bef-491c-bbd6-e67278f37987 Type: Storage >2014-08-06 07:39:56,697 INFO [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-48) [349f4a8e] Lock freed to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE >, sharedLocks= ] >2014-08-06 07:39:56,706 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-48) [349f4a8e] Correlation ID: 349f4a8e, Job ID: 7ad6da6d-ad2f-4aa1-9542-abddabcbf1c5, Call Stack: null, Custom Event ID: -1, Message: Storage Domain export (Data Center Default) was deactivated by system because it's not visible by any of the hosts. >2014-08-06 07:39:56,711 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-48) [349f4a8e] Removing vds [] from the domain in maintenance cache >2014-08-06 07:39:56,712 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-48) [349f4a8e] Removing host(s) [] from hosts unseen domain report cache >2014-08-06 07:39:59,822 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-36) [3e2ffa8] START, GetHardwareInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]), log id: 2ed48565 >2014-08-06 07:39:59,877 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-36) [3e2ffa8] FINISH, GetHardwareInfoVDSCommand, log id: 2ed48565 >2014-08-06 07:39:59,903 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-36) [455ca1b9] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS >2014-08-06 07:39:59,915 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-36) [5e70114d] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-08-06 07:39:59,924 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-36) [7ff3598b] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-08-06 07:39:59,936 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-36) [7ff3598b] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, storageType = NFS, connectionList = [{ id: 05047c9a-152e-4ef2-b118-4a1753c1eae0, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35export, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 6e1573a6 >2014-08-06 07:39:59,999 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-36) [7ff3598b] FINISH, ConnectStorageServerVDSCommand, return: {5083f370-af09-457b-9b73-3440e8b65a07=0, 05047c9a-152e-4ef2-b118-4a1753c1eae0=0}, log id: 6e1573a6 >2014-08-06 07:40:00,003 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-36) [7ff3598b] Host rack storage connection was succeeded >2014-08-06 07:40:00,006 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-97) [5c182324] Autorecovering 2 storage domains >2014-08-06 07:40:00,006 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-97) [5c182324] Autorecovering storage domains id: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 >2014-08-06 07:40:00,008 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-97) [1a0ac637] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage >2014-08-06 07:40:00,010 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-97) [1a0ac637] ConnectDomainToStorage. Before Connect all hosts to pool. Time:8/6/14 7:40 AM >2014-08-06 07:40:00,010 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-45) START, ConnectStoragePoolVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, masterVersion = 1), log id: 790cfb1 >2014-08-06 07:40:00,014 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-97) [1a0ac637] ConnectDomainToStorage. After Connect all hosts to pool. Time:8/6/14 7:40 AM >2014-08-06 07:40:00,017 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-97) [1a0ac637] Autorecovering storage domains id: 44559ec0-6bef-491c-bbd6-e67278f37987 >2014-08-06 07:40:00,019 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-97) [5d878efe] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 44559ec0-6bef-491c-bbd6-e67278f37987 Type: Storage >2014-08-06 07:40:00,021 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-97) [5d878efe] ConnectDomainToStorage. Before Connect all hosts to pool. Time:8/6/14 7:40 AM >2014-08-06 07:40:00,027 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-97) [5d878efe] ConnectDomainToStorage. After Connect all hosts to pool. Time:8/6/14 7:40 AM >2014-08-06 07:40:06,533 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-63) Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Inactive to Unknown, reason : no reporting hosts >2014-08-06 07:40:06,537 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-63) Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Inactive to Unknown, reason : no reporting hosts >2014-08-06 07:41:00,403 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-45) FINISH, ConnectStoragePoolVDSCommand, log id: 790cfb1 >2014-08-06 07:41:00,416 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-36) [7ff3598b] START, SetMOMPolicyParametersVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 51329ea5 >2014-08-06 07:41:00,512 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-36) [7ff3598b] FINISH, SetMOMPolicyParametersVDSCommand, log id: 51329ea5 >2014-08-06 07:41:00,519 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-36) [7ff3598b] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host rack was set to Up. >2014-08-06 07:41:00,525 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-36) [7ff3598b] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host rack. >2014-08-06 07:41:00,548 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-36) [405deed8] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS >2014-08-06 07:41:06,659 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-41) [5d8b5376] hostFromVds::selectedVds - rack, spmStatus Free, storage pool Default >2014-08-06 07:41:06,664 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-41) [5d8b5376] starting spm on vds rack, storage pool Default, prevId -1, LVER -1 >2014-08-06 07:41:06,666 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-41) [5d8b5376] START, SpmStartVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, prevId=-1, prevLVER=-1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 4ed27eb0 >2014-08-06 07:41:06,714 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-41) [5d8b5376] spmStart polling started: taskId = c31d6a09-0aa5-4fa1-a0fe-5a9a2de774fd >2014-08-06 07:41:07,721 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-41) [5d8b5376] spmStart polling ended: taskId = c31d6a09-0aa5-4fa1-a0fe-5a9a2de774fd task status = finished >2014-08-06 07:41:07,777 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-41) [5d8b5376] spmStart polling ended, spm status: SPM >2014-08-06 07:41:07,780 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-41) [5d8b5376] START, HSMClearTaskVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, taskId=c31d6a09-0aa5-4fa1-a0fe-5a9a2de774fd), log id: 1db71dce >2014-08-06 07:41:07,827 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-41) [5d8b5376] FINISH, HSMClearTaskVDSCommand, log id: 1db71dce >2014-08-06 07:41:07,828 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-41) [5d8b5376] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@41b78d10, log id: 4ed27eb0 >2014-08-06 07:41:07,831 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-41) [5d8b5376] Initialize Irs proxy from vds: 10.34.62.208 >2014-08-06 07:41:07,837 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-41) [5d8b5376] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208). >2014-08-06 07:41:07,846 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-40) [5d8b5376] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 76d15b2a >2014-08-06 07:41:07,881 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-40) [5d8b5376] -- executeIrsBrokerCommand: Attempting on storage pool 00000002-0002-0002-0002-0000000001d9 >2014-08-06 07:41:07,883 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-40) [5d8b5376] START, HSMGetAllTasksInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 236f6242 >2014-08-06 07:41:07,888 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-40) [5d8b5376] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 236f6242 >2014-08-06 07:41:07,889 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-40) [5d8b5376] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 76d15b2a >2014-08-06 07:41:07,890 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (org.ovirt.thread.pool-8-thread-40) [5d8b5376] Discovered no tasks on Storage Pool Default >2014-08-06 08:23:56,280 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-26) [6bd9692] Principal admin::internal synchronized >2014-08-06 08:24:25,360 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-85) [9016fac] Attempting to update VMs/Templates Ovf. >2014-08-06 09:23:56,293 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-50) [16080eb6] Principal admin::internal synchronized >2014-08-06 09:24:25,401 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-46) [56f1581c] Attempting to update VMs/Templates Ovf. >2014-08-06 09:56:02,856 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp--127.0.0.1-8702-10) Running command: LoginUserCommand internal: false. >2014-08-06 09:56:02,864 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-10) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-08-06 09:56:02,865 ERROR [org.ovirt.engine.core.aaa.filters.RestApiSessionMgmtFilter] (ajp--127.0.0.1-8702-10) Session-TTL header was not passed. Not setting TTL value >2014-08-06 09:59:32,378 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp--127.0.0.1-8702-4) Running command: LoginUserCommand internal: false. >2014-08-06 09:59:32,383 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-4) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-08-06 09:59:32,385 ERROR [org.ovirt.engine.core.aaa.filters.RestApiSessionMgmtFilter] (ajp--127.0.0.1-8702-4) Session-TTL header was not passed. Not setting TTL value >2014-08-06 10:01:05,911 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp--127.0.0.1-8702-1) Running command: LoginUserCommand internal: false. >2014-08-06 10:01:05,915 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-1) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-08-06 10:01:05,916 ERROR [org.ovirt.engine.core.aaa.filters.RestApiSessionMgmtFilter] (ajp--127.0.0.1-8702-1) Session-TTL header was not passed. Not setting TTL value >2014-08-06 10:02:13,431 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp--127.0.0.1-8702-11) Running command: LoginUserCommand internal: false. >2014-08-06 10:02:13,436 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-11) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-08-06 10:02:13,438 ERROR [org.ovirt.engine.core.aaa.filters.RestApiSessionMgmtFilter] (ajp--127.0.0.1-8702-11) Session-TTL header was not passed. Not setting TTL value >2014-08-06 10:04:42,129 INFO [org.ovirt.engine.core.bll.LoginAdminUserCommand] (ajp--127.0.0.1-8702-4) Running command: LoginAdminUserCommand internal: false. >2014-08-06 10:04:42,133 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-4) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-08-06 10:04:42,192 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp--127.0.0.1-8702-3) Running command: LoginUserCommand internal: false. >2014-08-06 10:07:23,428 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp--127.0.0.1-8702-2) Running command: LoginUserCommand internal: false. >2014-08-06 10:07:23,434 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-2) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-08-06 10:07:23,435 ERROR [org.ovirt.engine.core.aaa.filters.RestApiSessionMgmtFilter] (ajp--127.0.0.1-8702-2) Session-TTL header was not passed. Not setting TTL value >2014-08-06 10:10:24,724 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp--127.0.0.1-8702-2) Running command: LoginUserCommand internal: false. >2014-08-06 10:10:24,728 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-2) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-08-06 10:10:24,729 ERROR [org.ovirt.engine.core.aaa.filters.RestApiSessionMgmtFilter] (ajp--127.0.0.1-8702-2) Session-TTL header was not passed. Not setting TTL value >2014-08-06 10:12:16,226 ERROR [org.ovirt.engine.core.aaa.filters.RestApiSessionMgmtFilter] (ajp--127.0.0.1-8702-10) Session-TTL header was not passed. Not setting TTL value >2014-08-06 10:12:29,106 ERROR [org.ovirt.engine.core.aaa.filters.RestApiSessionMgmtFilter] (ajp--127.0.0.1-8702-1) Session-TTL header was not passed. Not setting TTL value >2014-08-06 10:12:29,153 ERROR [org.ovirt.engine.core.aaa.filters.RestApiSessionMgmtFilter] (ajp--127.0.0.1-8702-4) Session-TTL header was not passed. Not setting TTL value >2014-08-06 10:12:42,076 ERROR [org.ovirt.engine.core.aaa.filters.RestApiSessionMgmtFilter] (ajp--127.0.0.1-8702-5) Session-TTL header was not passed. Not setting TTL value >2014-08-06 10:12:42,115 ERROR [org.ovirt.engine.core.aaa.filters.RestApiSessionMgmtFilter] (ajp--127.0.0.1-8702-6) Session-TTL header was not passed. Not setting TTL value >2014-08-06 10:12:47,825 ERROR [org.ovirt.engine.core.aaa.filters.RestApiSessionMgmtFilter] (ajp--127.0.0.1-8702-3) Session-TTL header was not passed. Not setting TTL value >2014-08-06 10:12:47,895 ERROR [org.ovirt.engine.core.aaa.filters.RestApiSessionMgmtFilter] (ajp--127.0.0.1-8702-11) Session-TTL header was not passed. Not setting TTL value >2014-08-06 10:13:23,441 ERROR [org.ovirt.engine.core.aaa.filters.RestApiSessionMgmtFilter] (ajp--127.0.0.1-8702-10) Session-TTL header was not passed. Not setting TTL value >2014-08-06 10:13:23,479 ERROR [org.ovirt.engine.core.aaa.filters.RestApiSessionMgmtFilter] (ajp--127.0.0.1-8702-1) Session-TTL header was not passed. Not setting TTL value >2014-08-06 10:14:10,640 ERROR [org.ovirt.engine.core.aaa.filters.RestApiSessionMgmtFilter] (ajp--127.0.0.1-8702-6) Session-TTL header was not passed. Not setting TTL value >2014-08-06 10:14:10,681 ERROR [org.ovirt.engine.core.aaa.filters.RestApiSessionMgmtFilter] (ajp--127.0.0.1-8702-7) Session-TTL header was not passed. Not setting TTL value >2014-08-06 10:14:46,878 ERROR [org.ovirt.engine.core.aaa.filters.RestApiSessionMgmtFilter] (ajp--127.0.0.1-8702-11) Session-TTL header was not passed. Not setting TTL value >2014-08-06 10:14:46,918 ERROR [org.ovirt.engine.core.aaa.filters.RestApiSessionMgmtFilter] (ajp--127.0.0.1-8702-2) Session-TTL header was not passed. Not setting TTL value >2014-08-06 10:15:18,424 ERROR [org.ovirt.engine.core.aaa.filters.RestApiSessionMgmtFilter] (ajp--127.0.0.1-8702-1) Session-TTL header was not passed. Not setting TTL value >2014-08-06 10:15:18,460 ERROR [org.ovirt.engine.core.aaa.filters.RestApiSessionMgmtFilter] (ajp--127.0.0.1-8702-4) Session-TTL header was not passed. Not setting TTL value >2014-08-06 10:17:55,956 ERROR [org.ovirt.engine.core.aaa.filters.RestApiSessionMgmtFilter] (ajp--127.0.0.1-8702-10) Session-TTL header was not passed. Not setting TTL value >2014-08-06 10:17:55,992 ERROR [org.ovirt.engine.core.aaa.filters.RestApiSessionMgmtFilter] (ajp--127.0.0.1-8702-1) Session-TTL header was not passed. Not setting TTL value >2014-08-06 10:18:03,870 ERROR [org.ovirt.engine.core.aaa.filters.RestApiSessionMgmtFilter] (ajp--127.0.0.1-8702-4) Session-TTL header was not passed. Not setting TTL value >2014-08-06 10:18:03,907 ERROR [org.ovirt.engine.core.aaa.filters.RestApiSessionMgmtFilter] (ajp--127.0.0.1-8702-5) Session-TTL header was not passed. Not setting TTL value >2014-08-06 10:18:13,945 ERROR [org.ovirt.engine.core.aaa.filters.RestApiSessionMgmtFilter] (ajp--127.0.0.1-8702-6) Session-TTL header was not passed. Not setting TTL value >2014-08-06 10:18:13,987 ERROR [org.ovirt.engine.core.aaa.filters.RestApiSessionMgmtFilter] (ajp--127.0.0.1-8702-7) Session-TTL header was not passed. Not setting TTL value >2014-08-06 10:18:24,179 ERROR [org.ovirt.engine.core.aaa.filters.RestApiSessionMgmtFilter] (ajp--127.0.0.1-8702-3) Session-TTL header was not passed. Not setting TTL value >2014-08-06 10:18:24,220 ERROR [org.ovirt.engine.core.aaa.filters.RestApiSessionMgmtFilter] (ajp--127.0.0.1-8702-11) Session-TTL header was not passed. Not setting TTL value >2014-08-06 10:19:02,276 ERROR [org.ovirt.engine.core.aaa.filters.RestApiSessionMgmtFilter] (ajp--127.0.0.1-8702-1) Session-TTL header was not passed. Not setting TTL value >2014-08-06 10:19:03,059 ERROR [org.ovirt.engine.core.aaa.filters.RestApiSessionMgmtFilter] (ajp--127.0.0.1-8702-4) Session-TTL header was not passed. Not setting TTL value >2014-08-06 10:22:28,268 ERROR [org.ovirt.engine.core.aaa.filters.RestApiSessionMgmtFilter] (ajp--127.0.0.1-8702-10) Session-TTL header was not passed. Not setting TTL value >2014-08-06 10:22:28,305 ERROR [org.ovirt.engine.core.aaa.filters.RestApiSessionMgmtFilter] (ajp--127.0.0.1-8702-1) Session-TTL header was not passed. Not setting TTL value >2014-08-06 10:22:35,791 ERROR [org.ovirt.engine.core.aaa.filters.RestApiSessionMgmtFilter] (ajp--127.0.0.1-8702-4) Session-TTL header was not passed. Not setting TTL value >2014-08-06 10:22:35,828 ERROR [org.ovirt.engine.core.aaa.filters.RestApiSessionMgmtFilter] (ajp--127.0.0.1-8702-5) Session-TTL header was not passed. Not setting TTL value >2014-08-06 10:22:43,390 ERROR [org.ovirt.engine.core.aaa.filters.RestApiSessionMgmtFilter] (ajp--127.0.0.1-8702-7) Session-TTL header was not passed. Not setting TTL value >2014-08-06 10:22:43,432 ERROR [org.ovirt.engine.core.aaa.filters.RestApiSessionMgmtFilter] (ajp--127.0.0.1-8702-3) Session-TTL header was not passed. Not setting TTL value >2014-08-06 10:23:56,298 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-1) [795e8140] Principal admin::internal synchronized >2014-08-06 10:24:25,420 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-89) [5784753b] Attempting to update VMs/Templates Ovf. >2014-08-06 10:25:36,360 ERROR [org.ovirt.engine.core.aaa.filters.RestApiSessionMgmtFilter] (ajp--127.0.0.1-8702-5) Session-TTL header was not passed. Not setting TTL value >2014-08-06 10:25:36,405 ERROR [org.ovirt.engine.core.aaa.filters.RestApiSessionMgmtFilter] (ajp--127.0.0.1-8702-6) Session-TTL header was not passed. Not setting TTL value >2014-08-06 10:25:41,525 ERROR [org.ovirt.engine.core.aaa.filters.RestApiSessionMgmtFilter] (ajp--127.0.0.1-8702-7) Session-TTL header was not passed. Not setting TTL value >2014-08-06 10:25:41,570 ERROR [org.ovirt.engine.core.aaa.filters.RestApiSessionMgmtFilter] (ajp--127.0.0.1-8702-3) Session-TTL header was not passed. Not setting TTL value >2014-08-06 10:40:58,100 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-4) domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 in problem. vds: rack >2014-08-06 10:40:58,102 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-4) domain 44559ec0-6bef-491c-bbd6-e67278f37987:export in problem. vds: rack >2014-08-06 10:42:23,032 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-47) Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2014-08-06 10:42:23,036 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-47) [772ebc80] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-08-06 10:42:23,040 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-47) [772ebc80] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Active to Unknown, reason : null >2014-08-06 10:42:23,042 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-47) [772ebc80] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Active to Unknown, reason : null >2014-08-06 10:42:23,055 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-47) [772ebc80] Correlation ID: 772ebc80, Call Stack: null, Custom Event ID: -1, Message: Invalid status on Data Center Default. Setting Data Center status to Non Responsive (On host rack, Error: Cannot access master storage domain). >2014-08-06 10:44:23,083 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-47) [772ebc80] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2014-08-06 10:44:23,086 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-47) [772ebc80] hostFromVds::selectedVds - rack, spmStatus returned null! >2014-08-06 10:44:23,087 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-47) [772ebc80] IrsBroker::Failed::GetStoragePoolInfoVDS >2014-08-06 10:44:23,088 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-47) [772ebc80] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2014-08-06 10:44:23,124 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-2) [6607a054] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage >2014-08-06 10:44:23,141 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-2) [6607a054] START, ResetIrsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, ignoreStopFailed = true), log id: 1773d724 >2014-08-06 10:44:23,145 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-2) [6607a054] START, SpmStopVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9), log id: 164ca56d >2014-08-06 10:44:23,151 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-2) [6607a054] SpmStopVDSCommand::Stopping SPM on vds rack, pool id 00000002-0002-0002-0002-0000000001d9 >2014-08-06 10:45:00,009 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-57) [cd4c3ce] Autorecovering 1 storage domains >2014-08-06 10:45:00,010 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-57) [cd4c3ce] Autorecovering storage domains id: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 >2014-08-06 10:45:00,012 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-57) [a4c3498] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage >2014-08-06 10:45:00,014 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-57) [a4c3498] ConnectDomainToStorage. Before Connect all hosts to pool. Time:8/6/14 10:45 AM >2014-08-06 10:45:00,031 INFO [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-8-thread-10) [11260b8a] Running command: ConnectStorageToVdsCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2014-08-06 10:45:00,035 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-10) [11260b8a] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 4e1e615b >2014-08-06 10:46:23,061 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-73) [55405a19] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2014-08-06 10:46:23,064 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-73) [55405a19] hostFromVds::selectedVds - rack, spmStatus returned null! >2014-08-06 10:46:23,065 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-73) [55405a19] IrsBroker::Failed::GetStoragePoolInfoVDS >2014-08-06 10:46:23,066 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-73) [55405a19] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2014-08-06 10:47:23,064 WARN [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-2) [6607a054] could not stop spm of pool 00000002-0002-0002-0002-0000000001d9 on vds 17f7fe56-5f32-40ef-b098-611b84484e0b - reason: org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcRunTimeException: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException >2014-08-06 10:47:23,068 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ListVDSCommand] (DefaultQuartzScheduler_Worker-18) [5497a121] Command ListVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-08-06 10:47:23,072 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-18) [5497a121] Host rack is not responding. It will stay in Connecting state for a grace period of $80 seconds and after that an attempt to fence the host will be issued. >2014-08-06 10:47:23,066 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-10) [11260b8a] Command ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]) execution failed. Exception: VDSNetworkException: org.apache.commons.httpclient.NoHttpResponseException: The server 10.34.62.208 failed to respond >2014-08-06 10:47:23,079 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-10) [11260b8a] FINISH, ConnectStorageServerVDSCommand, log id: 4e1e615b >2014-08-06 10:47:23,071 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-54) [75c8e759] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-08-06 10:47:23,081 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-54) [75c8e759] hostFromVds::selectedVds - rack, spmStatus returned null! >2014-08-06 10:47:23,070 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-2) [6607a054] FINISH, SpmStopVDSCommand, log id: 164ca56d >2014-08-06 10:47:23,080 ERROR [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-8-thread-10) [11260b8a] Command org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: org.apache.commons.httpclient.NoHttpResponseException: The server 10.34.62.208 failed to respond (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-08-06 10:47:23,074 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-18) [5497a121] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:406) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.fetchRunningVms(VdsUpdateRunTimeInfo.java:971) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVmStats(VdsUpdateRunTimeInfo.java:920) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:638) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:235) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) [:1.7.0_55] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_55] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_55] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_55] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_55] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_55] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_55] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_55] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy82.list(Unknown Source) > at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source) [:1.7.0_55] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_55] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_55] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_55] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_55] > >2014-08-06 10:47:23,090 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-2) [6607a054] FINISH, ResetIrsVDSCommand, log id: 1773d724 >2014-08-06 10:47:23,088 ERROR [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-8-thread-10) [11260b8a] Transaction rolled-back for command: org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand. >2014-08-06 10:47:23,111 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-57) [a4c3498] ConnectDomainToStorage. After Connect all hosts to pool. Time:8/6/14 10:47 AM >2014-08-06 10:47:23,118 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-18) [5497a121] Failed to refresh VDS , vds = 17f7fe56-5f32-40ef-b098-611b84484e0b : rack, VDS Network Error, continuing. >java.net.ConnectException: Connection refused >2014-08-06 10:47:23,122 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-2) [6607a054] Correlation ID: 6607a054, Job ID: 34dd04ba-c423-4b37-a0eb-b57e499a5407, Call Stack: null, Custom Event ID: -1, Message: Failed to Reconstruct Master Domain for Data Center Default. >2014-08-06 10:47:23,129 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-26) [772ebc80] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue >2014-08-06 10:47:23,131 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-21) starting processDomainRecovery for domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 >2014-08-06 10:47:23,189 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-21) Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 was reported by all hosts in status UP as problematic. Not moving the domain to NonOperational because it is being reconstructed now. >2014-08-06 10:47:23,207 WARN [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-21) [5b35c6b2] CanDoAction of action ReconstructMasterDomain failed. Reasons:VAR__ACTION__RECONSTRUCT_MASTER,VAR__TYPE__STORAGE__DOMAIN,ACTION_TYPE_FAILED_NO_VDS_IN_POOL >2014-08-06 10:47:23,214 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-21) [5b35c6b2] Removing vds [] from the domain in maintenance cache >2014-08-06 10:47:23,215 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-21) [5b35c6b2] Removing host(s) [] from hosts unseen domain report cache >2014-08-06 10:47:23,216 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-26) [772ebc80] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue >2014-08-06 10:47:23,218 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-49) starting processDomainRecovery for domain 44559ec0-6bef-491c-bbd6-e67278f37987:export >2014-08-06 10:47:23,223 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-49) Domain 44559ec0-6bef-491c-bbd6-e67278f37987:export was reported by all hosts in status UP as problematic. Moving the domain to NonOperational. >2014-08-06 10:47:23,237 INFO [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-49) [582fadb7] Lock Acquired to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE >, sharedLocks= ] >2014-08-06 10:47:23,244 INFO [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-49) [582fadb7] Running command: DeactivateStorageDomainCommand internal: true. Entities affected : ID: 44559ec0-6bef-491c-bbd6-e67278f37987 Type: Storage >2014-08-06 10:47:23,249 INFO [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-49) [582fadb7] Lock freed to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE >, sharedLocks= ] >2014-08-06 10:47:23,258 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-49) [582fadb7] Correlation ID: 582fadb7, Job ID: ad6a13d7-d0d7-458d-987b-7cd154bcc4eb, Call Stack: null, Custom Event ID: -1, Message: Storage Domain export (Data Center Default) was deactivated by system because it's not visible by any of the hosts. >2014-08-06 10:47:23,263 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-49) [582fadb7] Removing vds [] from the domain in maintenance cache >2014-08-06 10:47:23,264 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-49) [582fadb7] Removing host(s) [] from hosts unseen domain report cache >2014-08-06 10:47:26,368 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-80) [16f4cea2] START, GetHardwareInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]), log id: 1a84ce87 >2014-08-06 10:47:26,420 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-80) [16f4cea2] FINISH, GetHardwareInfoVDSCommand, log id: 1a84ce87 >2014-08-06 10:47:26,444 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-80) [3b14ddd1] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS >2014-08-06 10:47:26,495 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-80) [6528240] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-08-06 10:47:26,502 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-80) [18c8a7ba] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-08-06 10:47:26,509 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-80) [18c8a7ba] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, storageType = NFS, connectionList = [{ id: 05047c9a-152e-4ef2-b118-4a1753c1eae0, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35export, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 3ea8369c >2014-08-06 10:47:26,569 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-80) [18c8a7ba] FINISH, ConnectStorageServerVDSCommand, return: {5083f370-af09-457b-9b73-3440e8b65a07=0, 05047c9a-152e-4ef2-b118-4a1753c1eae0=0}, log id: 3ea8369c >2014-08-06 10:47:26,570 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-80) [18c8a7ba] Host rack storage connection was succeeded >2014-08-06 10:47:26,578 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-19) START, ConnectStoragePoolVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, masterVersion = 1), log id: 16bfcf72 >2014-08-06 10:47:33,087 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-76) [5b2cf9d8] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Inactive to Unknown, reason : no reporting hosts >2014-08-06 10:47:33,090 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-76) [5b2cf9d8] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Inactive to Unknown, reason : no reporting hosts >2014-08-06 10:48:26,959 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-19) FINISH, ConnectStoragePoolVDSCommand, log id: 16bfcf72 >2014-08-06 10:48:26,975 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-80) [18c8a7ba] START, SetMOMPolicyParametersVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 15ef9e71 >2014-08-06 10:48:27,071 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-80) [18c8a7ba] FINISH, SetMOMPolicyParametersVDSCommand, log id: 15ef9e71 >2014-08-06 10:48:27,078 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-80) [18c8a7ba] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host rack was set to Up. >2014-08-06 10:48:27,085 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-80) [18c8a7ba] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host rack. >2014-08-06 10:48:27,099 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-80) [6b82e961] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS >2014-08-06 10:48:33,250 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-82) [3e9bda56] hostFromVds::selectedVds - rack, spmStatus Free, storage pool Default >2014-08-06 10:48:33,256 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-82) [3e9bda56] starting spm on vds rack, storage pool Default, prevId -1, LVER -1 >2014-08-06 10:48:33,258 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-82) [3e9bda56] START, SpmStartVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, prevId=-1, prevLVER=-1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 28679213 >2014-08-06 10:48:33,306 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-82) [3e9bda56] spmStart polling started: taskId = 28b2276c-2810-4e5a-af1f-f30a1918032a >2014-08-06 10:48:34,313 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-82) [3e9bda56] spmStart polling ended: taskId = 28b2276c-2810-4e5a-af1f-f30a1918032a task status = finished >2014-08-06 10:48:34,369 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-82) [3e9bda56] spmStart polling ended, spm status: SPM >2014-08-06 10:48:34,371 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-82) [3e9bda56] START, HSMClearTaskVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, taskId=28b2276c-2810-4e5a-af1f-f30a1918032a), log id: 4c991eb6 >2014-08-06 10:48:34,417 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-82) [3e9bda56] FINISH, HSMClearTaskVDSCommand, log id: 4c991eb6 >2014-08-06 10:48:34,417 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-82) [3e9bda56] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@545f4b93, log id: 28679213 >2014-08-06 10:48:34,421 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-82) [3e9bda56] Initialize Irs proxy from vds: 10.34.62.208 >2014-08-06 10:48:34,428 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-82) [3e9bda56] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208). >2014-08-06 10:48:34,440 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-37) [3e9bda56] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: e685a49 >2014-08-06 10:48:34,470 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-37) [3e9bda56] -- executeIrsBrokerCommand: Attempting on storage pool 00000002-0002-0002-0002-0000000001d9 >2014-08-06 10:48:34,473 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-37) [3e9bda56] START, HSMGetAllTasksInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 5970fa9d >2014-08-06 10:48:34,479 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-37) [3e9bda56] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 5970fa9d >2014-08-06 10:48:34,481 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-37) [3e9bda56] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: e685a49 >2014-08-06 10:48:34,482 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (org.ovirt.thread.pool-8-thread-37) [3e9bda56] Discovered no tasks on Storage Pool Default >2014-08-06 11:23:56,304 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-82) [3e9bda56] Principal admin::internal synchronized >2014-08-06 11:24:25,441 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-46) [56f1581c] Attempting to update VMs/Templates Ovf. >2014-08-06 11:24:26,746 WARN [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-8-thread-26) Executing a command: java.util.concurrent.FutureTask , but note that there are 1 tasks in the queue. >2014-08-06 12:23:56,310 INFO [org.ovirt.engine.core.bll.SyncUsers] (DefaultQuartzScheduler_Worker-79) [37510499] Principal admin::internal synchronized >2014-08-06 12:24:25,457 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-63) Attempting to update VMs/Templates Ovf. >2014-08-06 12:30:52,970 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Loaded file "/usr/share/ovirt-engine/services/ovirt-engine/ovirt-engine.conf". >2014-08-06 12:30:52,976 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) The file "/etc/ovirt-engine/engine.conf" doesn't exist or isn't readable. Will return an empty set of properties. >2014-08-06 12:30:52,977 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Loaded file "/etc/ovirt-engine/engine.conf.d/10-setup-database.conf". >2014-08-06 12:30:52,977 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Loaded file "/etc/ovirt-engine/engine.conf.d/10-setup-jboss.conf". >2014-08-06 12:30:52,980 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Loaded file "/etc/ovirt-engine/engine.conf.d/10-setup-pki.conf". >2014-08-06 12:30:52,980 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Loaded file "/etc/ovirt-engine/engine.conf.d/10-setup-protocols.conf". >2014-08-06 12:30:52,981 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_AJP_ENABLED" is "true". >2014-08-06 12:30:52,982 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_AJP_PORT" is "8702". >2014-08-06 12:30:52,982 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_APPS" is "engine.ear restapi.war legacy_restapi.war". >2014-08-06 12:30:52,983 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_CACHE" is "/var/cache/ovirt-engine". >2014-08-06 12:30:52,983 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_DB_CHECK_INTERVAL" is "1000". >2014-08-06 12:30:52,990 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_DB_CONNECTION_TIMEOUT" is "300000". >2014-08-06 12:30:52,990 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_DB_DATABASE" is "engine". >2014-08-06 12:30:52,991 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_DB_DRIVER" is "org.postgresql.Driver". >2014-08-06 12:30:52,991 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_DB_HOST" is "localhost". >2014-08-06 12:30:52,992 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_DB_MAX_CONNECTIONS" is "100". >2014-08-06 12:30:52,992 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_DB_MIN_CONNECTIONS" is "1". >2014-08-06 12:30:52,993 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_DB_PASSWORD" is "***". >2014-08-06 12:30:52,993 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_DB_PORT" is "5432". >2014-08-06 12:30:52,994 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_DB_SECURED" is "False". >2014-08-06 12:30:52,994 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_DB_SECURED_VALIDATION" is "False". >2014-08-06 12:30:52,995 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_DB_URL" is "jdbc:postgresql://localhost:5432/engine?sslfactory=org.postgresql.ssl.NonValidatingFactory". >2014-08-06 12:30:53,002 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_DB_USER" is "engine". >2014-08-06 12:30:53,002 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_DEBUG_ADDRESS" is "". >2014-08-06 12:30:53,003 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_DOC" is "/usr/share/doc/ovirt-engine". >2014-08-06 12:30:53,003 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_ETC" is "/etc/ovirt-engine". >2014-08-06 12:30:53,004 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_EXTENSION_PATH" is "/usr/share/ovirt-engine/extensions.d:/etc/ovirt-engine/extensions.d". >2014-08-06 12:30:53,004 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_FQDN" is "pb-rh35.rhev.lab.eng.brq.redhat.com". >2014-08-06 12:30:53,005 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_GROUP" is "ovirt". >2014-08-06 12:30:53,005 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_HEAP_MAX" is "1g". >2014-08-06 12:30:53,006 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_HEAP_MIN" is "1g". >2014-08-06 12:30:53,006 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_HTTPS_ENABLED" is "false". >2014-08-06 12:30:53,007 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_HTTPS_PORT" is "None". >2014-08-06 12:30:53,007 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_HTTPS_PROTOCOLS" is "SSLv3,TLSv1,TLSv1.1,TLSv1.2". >2014-08-06 12:30:53,014 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_HTTP_ENABLED" is "false". >2014-08-06 12:30:53,014 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_HTTP_PORT" is "None". >2014-08-06 12:30:53,015 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_JAVA_MODULEPATH" is "/usr/share/ovirt-engine/modules". >2014-08-06 12:30:53,015 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_JVM_ARGS" is " -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath="/var/log/ovirt-engine/dump"". >2014-08-06 12:30:53,016 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_LOG" is "/var/log/ovirt-engine". >2014-08-06 12:30:53,016 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_LOG_TO_CONSOLE" is "false". >2014-08-06 12:30:53,017 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_MANUAL" is "/usr/share/ovirt-engine/manual". >2014-08-06 12:30:53,017 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_PERM_MAX" is "256m". >2014-08-06 12:30:53,018 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_PERM_MIN" is "256m". >2014-08-06 12:30:53,018 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_PKI" is "/etc/pki/ovirt-engine". >2014-08-06 12:30:53,019 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_PKI_CA" is "/etc/pki/ovirt-engine/ca.pem". >2014-08-06 12:30:53,019 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_PKI_ENGINE_CERT" is "/etc/pki/ovirt-engine/certs/engine.cer". >2014-08-06 12:30:53,022 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_PKI_ENGINE_STORE" is "/etc/pki/ovirt-engine/keys/engine.p12". >2014-08-06 12:30:53,023 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_PKI_ENGINE_STORE_ALIAS" is "1". >2014-08-06 12:30:53,023 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_PKI_ENGINE_STORE_PASSWORD" is "***". >2014-08-06 12:30:53,024 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_PKI_TRUST_STORE" is "/etc/pki/ovirt-engine/.truststore". >2014-08-06 12:30:53,029 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_PKI_TRUST_STORE_PASSWORD" is "***". >2014-08-06 12:30:53,030 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_PROPERTIES" is " jsse.enableSNIExtension=false "java.security.krb5.conf=/etc/ovirt-engine/krb5.conf"". >2014-08-06 12:30:53,030 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_PROXY_ENABLED" is "true". >2014-08-06 12:30:53,031 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_PROXY_HTTPS_PORT" is "443". >2014-08-06 12:30:53,032 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_PROXY_HTTP_PORT" is "80". >2014-08-06 12:30:53,032 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_REPORTS_UI" is "/var/lib/ovirt-engine/reports.xml". >2014-08-06 12:30:53,033 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_STOP_INTERVAL" is "1". >2014-08-06 12:30:53,033 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_STOP_TIME" is "10". >2014-08-06 12:30:53,033 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_TMP" is "/var/tmp/ovirt-engine". >2014-08-06 12:30:53,034 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_UP_MARK" is "/var/lib/ovirt-engine/engine.up". >2014-08-06 12:30:53,035 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_URI" is "/ovirt-engine". >2014-08-06 12:30:53,036 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_USER" is "ovirt". >2014-08-06 12:30:53,036 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_USR" is "/usr/share/ovirt-engine". >2014-08-06 12:30:53,037 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_VAR" is "/var/lib/ovirt-engine". >2014-08-06 12:30:53,040 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_VERBOSE_GC" is "false". >2014-08-06 12:30:53,041 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "JBOSS_HOME" is "/usr/share/ovirt-engine-jboss-as". >2014-08-06 12:30:53,042 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "JBOSS_RUNTIME" is "/var/lib/ovirt-engine/jboss_runtime". >2014-08-06 12:30:53,043 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "SENSITIVE_KEYS" is ",ENGINE_DB_PASSWORD,ENGINE_PKI_TRUST_STORE_PASSWORD,ENGINE_PKI_ENGINE_STORE_PASSWORD". >2014-08-06 12:30:53,737 INFO [org.ovirt.engine.core.bll.Backend] (MSC service thread 1-4) Start initializing Backend >2014-08-06 12:30:54,203 INFO [org.ovirt.engine.core.utils.osinfo.OsInfoPreferencesLoader] (MSC service thread 1-4) Loaded file /etc/ovirt-engine/osinfo.conf.d/00-defaults.properties >2014-08-06 12:30:54,311 INFO [org.ovirt.engine.core.bll.Backend] (MSC service thread 1-4) Running ovirt-engine 3.5.0-0.0.master.20140804172041.git23b558e.el6 >2014-08-06 12:30:54,311 INFO [org.ovirt.engine.core.bll.CpuFlagsManagerHandler] (MSC service thread 1-4) Start initializing dictionaries >2014-08-06 12:30:54,314 INFO [org.ovirt.engine.core.bll.CpuFlagsManagerHandler] (MSC service thread 1-4) Finished initializing dictionaries >2014-08-06 12:30:54,315 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (MSC service thread 1-4) Start initializing AuditLogCleanupManager >2014-08-06 12:30:54,321 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (MSC service thread 1-4) Setting audit cleanup manager to run at: 35 35 3 * * ? >2014-08-06 12:30:54,355 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (MSC service thread 1-4) Finished initializing AuditLogCleanupManager >2014-08-06 12:30:54,356 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (MSC service thread 1-4) Start initializing CommandEntityCleanupManager >2014-08-06 12:30:54,358 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (MSC service thread 1-4) Setting command entity cleanup manager to run at: 35 35 3 * * ? >2014-08-06 12:30:54,363 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (MSC service thread 1-4) Finished initializing CommandEntityCleanupManager >2014-08-06 12:30:54,365 INFO [org.ovirt.engine.core.bll.TagsDirector] (MSC service thread 1-4) Start initializing TagsDirector >2014-08-06 12:30:54,374 INFO [org.ovirt.engine.core.bll.TagsDirector] (MSC service thread 1-4) Tag root added to tree >2014-08-06 12:30:54,385 INFO [org.ovirt.engine.core.bll.TagsDirector] (MSC service thread 1-4) Finished initializing TagsDirector >2014-08-06 12:30:54,386 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (MSC service thread 1-4) Start initializing IsoDomainListSyncronizer >2014-08-06 12:30:54,393 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (MSC service thread 1-4) Finished initializing IsoDomainListSyncronizer >2014-08-06 12:30:54,517 INFO [org.ovirt.engine.core.bll.Backend] (MSC service thread 1-4) Completed initializing handlers >2014-08-06 12:30:54,544 INFO [org.ovirt.engine.core.utils.ErrorTranslatorImpl] (MSC service thread 1-4) Start initializing ErrorTranslatorImpl >2014-08-06 12:30:54,563 WARN [org.ovirt.engine.core.utils.ErrorTranslatorImpl] (MSC service thread 1-4) Code MAC_ADDRESS_IS_IN_USE appears more than once in string table. >2014-08-06 12:30:54,564 INFO [org.ovirt.engine.core.utils.ErrorTranslatorImpl] (MSC service thread 1-4) Finished initializing ErrorTranslatorImpl >2014-08-06 12:30:54,565 INFO [org.ovirt.engine.core.utils.ErrorTranslatorImpl] (MSC service thread 1-4) Start initializing ErrorTranslatorImpl >2014-08-06 12:30:54,565 INFO [org.ovirt.engine.core.utils.ErrorTranslatorImpl] (MSC service thread 1-4) Finished initializing ErrorTranslatorImpl >2014-08-06 12:30:54,566 INFO [org.ovirt.engine.core.bll.Backend] (MSC service thread 1-4) Mark incomplete jobs as UNKNOWN >2014-08-06 12:30:54,594 INFO [org.ovirt.engine.core.bll.job.JobRepositoryCleanupManager] (MSC service thread 1-4) Start initializing JobRepositoryCleanupManager >2014-08-06 12:30:54,595 INFO [org.ovirt.engine.core.bll.job.JobRepositoryCleanupManager] (MSC service thread 1-4) Finished initializing JobRepositoryCleanupManager >2014-08-06 12:30:54,595 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (MSC service thread 1-4) Start initializing AutoRecoveryManager >2014-08-06 12:30:54,597 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (MSC service thread 1-4) Finished initializing AutoRecoveryManager >2014-08-06 12:30:54,599 INFO [org.ovirt.engine.core.dal.job.ExecutionMessageDirector] (MSC service thread 1-4) Start initializing ExecutionMessageDirector >2014-08-06 12:30:54,601 INFO [org.ovirt.engine.core.dal.job.ExecutionMessageDirector] (MSC service thread 1-4) Finished initializing ExecutionMessageDirector >2014-08-06 12:30:54,656 INFO [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (MSC service thread 1-4) Start of enabled extensions list >2014-08-06 12:30:54,657 INFO [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (MSC service thread 1-4) Instance name: 'builtin-authn-internal', Extension name: 'Internal Authn (Built-in)', Version: 'N/A', Notes: '', License: 'ASL 2.0', Home: 'http://www.ovirt.org', Author 'The oVirt Project', Build interface Version: '0', File: 'N/A', Initialized: 'true' >2014-08-06 12:30:54,658 INFO [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (MSC service thread 1-4) Instance name: 'internal', Extension name: 'Internal Authz (Built-in)', Version: 'N/A', Notes: '', License: 'ASL 2.0', Home: 'http://www.ovirt.org', Author 'The oVirt Project', Build interface Version: '0', File: 'N/A', Initialized: 'true' >2014-08-06 12:30:54,659 INFO [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (MSC service thread 1-4) End of enabled extensions list >2014-08-06 12:30:54,747 INFO [org.ovirt.engine.core.bll.aaa.DbUserCacheManager] (MSC service thread 1-1) Start initializing DbUserCacheManager >2014-08-06 12:30:54,756 INFO [org.ovirt.engine.core.bll.aaa.DbUserCacheManager] (MSC service thread 1-1) Finished initializing DbUserCacheManager >2014-08-06 12:30:54,780 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (MSC service thread 1-1) Initialization of AsyncTaskManager completed successfully. >2014-08-06 12:30:54,785 INFO [org.ovirt.engine.core.vdsbroker.ResourceManager] (MSC service thread 1-1) Start initializing ResourceManager >2014-08-06 12:30:54,805 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-1) Principal admin::internal synchronized >2014-08-06 12:30:54,877 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (MSC service thread 1-1) Entered VdsManager constructor >2014-08-06 12:30:54,988 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (MSC service thread 1-1) Initialize vdsBroker (10.34.62.208,54,321) >2014-08-06 12:30:55,205 INFO [org.ovirt.engine.core.vdsbroker.ResourceManager] (MSC service thread 1-1) VDS 17f7fe56-5f32-40ef-b098-611b84484e0b was added to the Resource Manager >2014-08-06 12:30:55,254 INFO [org.ovirt.engine.core.vdsbroker.ResourceManager] (MSC service thread 1-1) Finished initializing ResourceManager >2014-08-06 12:30:55,264 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (MSC service thread 1-1) Initialization of OvfDataUpdater completed successfully. >2014-08-06 12:30:55,265 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (MSC service thread 1-1) Start scheduling to enable vds load balancer >2014-08-06 12:30:55,266 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (MSC service thread 1-1) Finished scheduling to enable vds load balancer >2014-08-06 12:30:55,267 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (MSC service thread 1-1) Start HA Reservation check >2014-08-06 12:30:55,268 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (MSC service thread 1-1) Finished HA Reservation check >2014-08-06 12:30:55,283 INFO [org.ovirt.engine.core.bll.network.MacPoolManagerRanges] (org.ovirt.thread.pool-8-thread-1) Start initializing MacPoolManagerRanges >2014-08-06 12:30:55,291 INFO [org.ovirt.engine.core.bll.InitBackendServicesOnStartupBean] (MSC service thread 1-1) Init VM custom properties utilities >2014-08-06 12:30:55,292 INFO [org.ovirt.engine.core.bll.InitBackendServicesOnStartupBean] (MSC service thread 1-1) Init device custom properties utilities >2014-08-06 12:30:55,307 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (MSC service thread 1-1) Initializing Scheduling manager >2014-08-06 12:30:55,320 INFO [org.ovirt.engine.core.bll.network.MacPoolManagerRanges] (org.ovirt.thread.pool-8-thread-1) Finished initializing. Available MACs in pool: 253 >2014-08-06 12:30:55,376 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (MSC service thread 1-1) External scheduler disabled, discovery skipped >2014-08-06 12:30:55,377 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (MSC service thread 1-1) Initialized Scheduling manager >2014-08-06 12:30:55,378 INFO [org.ovirt.engine.core.bll.dwh.DwhHeartBeat] (MSC service thread 1-1) Initializing DWH Heart Beat >2014-08-06 12:30:55,385 INFO [org.ovirt.engine.core.bll.dwh.DwhHeartBeat] (MSC service thread 1-1) DWH Heart Beat initialized >2014-08-06 12:30:58,932 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-6) START, GetHardwareInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]), log id: ef8cbb >2014-08-06 12:30:58,999 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-6) FINISH, GetHardwareInfoVDSCommand, log id: ef8cbb >2014-08-06 12:30:59,284 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-6) Initializing Host: rack >2014-08-06 12:30:59,450 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-6) [17385311] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS >2014-08-06 12:31:05,354 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-19) [6c6ade78] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-08-06 12:31:05,379 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-19) [6c6ade78] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Active to Unknown, reason : null >2014-08-06 12:31:05,395 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-19) [6c6ade78] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Active to Unknown, reason : null >2014-08-06 12:31:05,419 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-19) [6c6ade78] Cannot convert the string Host ${VdsName} is not responding. It will stay in Connecting state for a grace period of ${Seconds} seconds and after that an attempt to fence the host will be issued. to AuditLogType, the key does not exist in the AuditLogType declared types >2014-08-06 12:31:05,432 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-19) [6c6ade78] AuditLogType: UNASSIGNED not exist in string table >2014-08-06 12:31:05,433 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-19) [6c6ade78] AuditLogType: VDS_AUTO_FENCE_STATUS not exist in string table >2014-08-06 12:31:05,434 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-19) [6c6ade78] AuditLogType: VDS_AUTO_FENCE_STATUS_FAILED not exist in string table >2014-08-06 12:31:05,435 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-19) [6c6ade78] AuditLogType: VDS_AUTO_FENCE_FAILED_CALL_FENCE_SPM not exist in string table >2014-08-06 12:31:05,436 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-19) [6c6ade78] AuditLogType: VDS_HIGH_NETWORK_USE not exist in string table >2014-08-06 12:31:05,437 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-19) [6c6ade78] AuditLogType: USER_FAILED_REMOVE_VM not exist in string table >2014-08-06 12:31:05,438 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-19) [6c6ade78] AuditLogType: USER_RUN_UNLOCK_ENTITY_SCRIPT not exist in string table >2014-08-06 12:31:05,440 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-19) [6c6ade78] AuditLogType: VDS_NETWORK_MTU_DIFFER_FROM_LOGICAL_NETWORK not exist in string table >2014-08-06 12:31:05,441 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-19) [6c6ade78] AuditLogType: VDS_HOST_IN_CONNECTING_STATE not exist in string table >2014-08-06 12:31:05,442 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-19) [6c6ade78] AuditLogType: STORAGE_ACTIVATE_ASYNC not exist in string table >2014-08-06 12:31:05,446 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-19) [6c6ade78] AuditLogType: DWH_STOPPED not exist in string table >2014-08-06 12:31:05,447 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-19) [6c6ade78] AuditLogType: DWH_STARTED not exist in string table >2014-08-06 12:31:05,448 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-19) [6c6ade78] AuditLogType: DWH_ERROR not exist in string table >2014-08-06 12:31:05,450 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-19) [6c6ade78] AuditLogType: USER_REMOVE_AUDIT_LOG not exist in string table >2014-08-06 12:31:05,451 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-19) [6c6ade78] AuditLogType: USER_REMOVE_AUDIT_LOG_FAILED not exist in string table >2014-08-06 12:31:05,458 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-19) [6c6ade78] AuditLogType: USER_CLEAR_ALL_DISMISSED_AUDIT_LOG not exist in string table >2014-08-06 12:31:05,459 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-19) [6c6ade78] AuditLogType: USER_CLEAR_ALL_DISMISSED_AUDIT_LOG_FAILED not exist in string table >2014-08-06 12:31:05,798 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-19) [6c6ade78] Correlation ID: 6c6ade78, Call Stack: null, Custom Event ID: -1, Message: Invalid status on Data Center Default. Setting status to Non Responsive. >2014-08-06 12:31:05,935 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-19) [6c6ade78] hostFromVds::selectedVds - rack, spmStatus SPM, storage pool Default >2014-08-06 12:31:05,947 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-19) [6c6ade78] Initialize Irs proxy from vds: 10.34.62.208 >2014-08-06 12:31:05,955 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-19) [6c6ade78] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208). >2014-08-06 12:31:05,978 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-15) [6c6ade78] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 161f87e5 >2014-08-06 12:31:06,027 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-15) [6c6ade78] -- executeIrsBrokerCommand: Attempting on storage pool 00000002-0002-0002-0002-0000000001d9 >2014-08-06 12:31:06,035 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-15) [6c6ade78] START, HSMGetAllTasksInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 72f183c3 >2014-08-06 12:31:06,042 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-15) [6c6ade78] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 72f183c3 >2014-08-06 12:31:06,043 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-15) [6c6ade78] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 161f87e5 >2014-08-06 12:31:06,044 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (org.ovirt.thread.pool-8-thread-15) [6c6ade78] Discovered no tasks on Storage Pool Default >2014-08-06 12:31:48,319 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-1) Running command: LoginUserCommand internal: false. >2014-08-06 12:31:48,329 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-1) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-08-06 13:14:56,938 WARN [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-8-thread-1) Executing a command: java.util.concurrent.FutureTask , but note that there are 0 tasks in the queue. >2014-08-06 13:15:12,074 WARN [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-8-thread-6) Executing a command: java.util.concurrent.FutureTask , but note that there are 0 tasks in the queue. >2014-08-06 13:30:54,829 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-46) Principal admin::internal synchronized >2014-08-06 13:30:55,264 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-53) Attempting to update VMs/Templates Ovf. >2014-08-06 13:48:14,475 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-7) domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 in problem. vds: rack >2014-08-06 13:48:29,604 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-32) domain 44559ec0-6bef-491c-bbd6-e67278f37987:export in problem. vds: rack >2014-08-06 13:49:54,054 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-58) Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2014-08-06 13:49:54,074 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-58) [ba1ce2a] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-08-06 13:49:54,085 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-58) [ba1ce2a] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Active to Unknown, reason : null >2014-08-06 13:49:54,087 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-58) [ba1ce2a] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Active to Unknown, reason : null >2014-08-06 13:49:54,103 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-58) [ba1ce2a] Correlation ID: ba1ce2a, Call Stack: null, Custom Event ID: -1, Message: Invalid status on Data Center Default. Setting Data Center status to Non Responsive (On host rack, Error: Cannot access master storage domain). >2014-08-06 13:51:54,110 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-58) [ba1ce2a] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2014-08-06 13:51:54,113 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-58) [ba1ce2a] hostFromVds::selectedVds - rack, spmStatus returned null! >2014-08-06 13:51:54,115 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-58) [ba1ce2a] IrsBroker::Failed::GetStoragePoolInfoVDS >2014-08-06 13:51:54,116 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-58) [ba1ce2a] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2014-08-06 13:51:54,234 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-5) [663debf6] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage >2014-08-06 13:51:54,307 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-5) [663debf6] START, ResetIrsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, ignoreStopFailed = true), log id: 1e2cce75 >2014-08-06 13:51:54,313 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-5) [663debf6] START, SpmStopVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9), log id: 5a24a037 >2014-08-06 13:51:54,320 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-5) [663debf6] SpmStopVDSCommand::Stopping SPM on vds rack, pool id 00000002-0002-0002-0002-0000000001d9 >2014-08-06 13:53:54,080 WARN [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-5) [663debf6] could not stop spm of pool 00000002-0002-0002-0002-0000000001d9 on vds 17f7fe56-5f32-40ef-b098-611b84484e0b - reason: org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcRunTimeException: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException >2014-08-06 13:53:54,082 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-5) [663debf6] FINISH, SpmStopVDSCommand, log id: 5a24a037 >2014-08-06 13:53:54,089 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-5) [663debf6] FINISH, ResetIrsVDSCommand, log id: 1e2cce75 >2014-08-06 13:53:54,093 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ListVDSCommand] (DefaultQuartzScheduler_Worker-74) Command ListVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-08-06 13:53:54,101 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-74) Host rack is not responding. It will stay in Connecting state for a grace period of $80 seconds and after that an attempt to fence the host will be issued. >2014-08-06 13:53:54,102 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-66) Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: VDSNetworkException: org.apache.commons.httpclient.NoHttpResponseException: The server 10.34.62.208 failed to respond >2014-08-06 13:53:54,106 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-66) hostFromVds::selectedVds - rack, spmStatus returned null! >2014-08-06 13:53:54,103 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-74) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:407) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.fetchRunningVms(VdsUpdateRunTimeInfo.java:971) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVmStats(VdsUpdateRunTimeInfo.java:920) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:638) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor17.invoke(Unknown Source) [:1.7.0_55] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_55] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_55] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_55] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_55] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_55] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_55] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_55] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.list(Unknown Source) > at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source) [:1.7.0_55] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_55] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_55] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_55] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_55] > >2014-08-06 13:53:54,148 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-74) Failed to refresh VDS , vds = 17f7fe56-5f32-40ef-b098-611b84484e0b : rack, VDS Network Error, continuing. >java.net.ConnectException: Connection refused >2014-08-06 13:53:54,149 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-5) [663debf6] Correlation ID: 663debf6, Job ID: 189806e5-27f6-490f-9007-79d6f6e94e18, Call Stack: null, Custom Event ID: -1, Message: Failed to Reconstruct Master Domain for Data Center Default. >2014-08-06 13:53:54,173 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-43) [ba1ce2a] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue >2014-08-06 13:53:54,176 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-15) starting processDomainRecovery for domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 >2014-08-06 13:53:54,245 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-15) Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 was reported by all hosts in status UP as problematic. Not moving the domain to NonOperational because it is being reconstructed now. >2014-08-06 13:53:54,272 WARN [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-15) [7e2fe5e7] CanDoAction of action ReconstructMasterDomain failed. Reasons:VAR__ACTION__RECONSTRUCT_MASTER,VAR__TYPE__STORAGE__DOMAIN,ACTION_TYPE_FAILED_NO_VDS_IN_POOL >2014-08-06 13:53:54,284 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-15) [7e2fe5e7] Removing vds [] from the domain in maintenance cache >2014-08-06 13:53:54,285 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-15) [7e2fe5e7] Removing host(s) [] from hosts unseen domain report cache >2014-08-06 13:53:54,286 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-43) [ba1ce2a] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue >2014-08-06 13:53:54,288 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-8) starting processDomainRecovery for domain 44559ec0-6bef-491c-bbd6-e67278f37987:export >2014-08-06 13:53:54,293 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-8) Domain 44559ec0-6bef-491c-bbd6-e67278f37987:export was reported by all hosts in status UP as problematic. Moving the domain to NonOperational. >2014-08-06 13:53:54,313 INFO [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-8) [7c8b8632] Lock Acquired to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE >, sharedLocks= ] >2014-08-06 13:53:54,323 INFO [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-8) [7c8b8632] Running command: DeactivateStorageDomainCommand internal: true. Entities affected : ID: 44559ec0-6bef-491c-bbd6-e67278f37987 Type: Storage >2014-08-06 13:53:54,350 INFO [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-8) [7c8b8632] Lock freed to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE >, sharedLocks= ] >2014-08-06 13:53:54,369 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-8) [7c8b8632] Correlation ID: 7c8b8632, Job ID: 898f2d4c-9716-4c8b-bbe0-9d7f5acc25bf, Call Stack: null, Custom Event ID: -1, Message: Storage Domain export (Data Center Default) was deactivated by system because it's not visible by any of the hosts. >2014-08-06 13:53:54,376 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-8) [7c8b8632] Removing vds [] from the domain in maintenance cache >2014-08-06 13:53:54,376 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-8) [7c8b8632] Removing host(s) [] from hosts unseen domain report cache >2014-08-06 13:53:57,455 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-14) START, GetHardwareInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]), log id: 14120542 >2014-08-06 13:53:57,508 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-14) FINISH, GetHardwareInfoVDSCommand, log id: 14120542 >2014-08-06 13:53:57,592 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-14) [34e4d0fb] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS >2014-08-06 13:53:57,640 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-14) [1908f1df] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-08-06 13:53:57,668 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-14) [61b8688e] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-08-06 13:53:57,694 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-14) [61b8688e] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, storageType = NFS, connectionList = [{ id: 05047c9a-152e-4ef2-b118-4a1753c1eae0, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35export, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 12e153fe >2014-08-06 13:53:57,724 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-14) [61b8688e] FINISH, ConnectStorageServerVDSCommand, return: {5083f370-af09-457b-9b73-3440e8b65a07=0, 05047c9a-152e-4ef2-b118-4a1753c1eae0=0}, log id: 12e153fe >2014-08-06 13:53:57,727 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-14) [61b8688e] Host rack storage connection was succeeded >2014-08-06 13:53:57,741 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-11) START, ConnectStoragePoolVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, masterVersion = 1), log id: 4ef7b96b >2014-08-06 13:54:04,119 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-11) Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Inactive to Unknown, reason : no reporting hosts >2014-08-06 13:54:04,124 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-11) Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Inactive to Unknown, reason : no reporting hosts >2014-08-06 13:54:58,141 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-11) FINISH, ConnectStoragePoolVDSCommand, log id: 4ef7b96b >2014-08-06 13:54:58,159 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-14) [61b8688e] START, SetMOMPolicyParametersVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 7eb8dd85 >2014-08-06 13:54:58,253 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-14) [61b8688e] FINISH, SetMOMPolicyParametersVDSCommand, log id: 7eb8dd85 >2014-08-06 13:54:58,261 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-14) [61b8688e] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host rack was set to Up. >2014-08-06 13:54:58,270 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-14) [61b8688e] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host rack. >2014-08-06 13:54:58,301 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-14) [6bdb6450] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS >2014-08-06 13:55:04,269 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-50) hostFromVds::selectedVds - rack, spmStatus Free, storage pool Default >2014-08-06 13:55:04,282 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-50) starting spm on vds rack, storage pool Default, prevId -1, LVER -1 >2014-08-06 13:55:04,285 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-50) START, SpmStartVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, prevId=-1, prevLVER=-1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 4f376c77 >2014-08-06 13:55:04,333 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-50) spmStart polling started: taskId = 9047e6c8-834e-4666-80c1-6158df8b31df >2014-08-06 13:55:05,342 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-50) spmStart polling ended: taskId = 9047e6c8-834e-4666-80c1-6158df8b31df task status = finished >2014-08-06 13:55:05,398 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-50) spmStart polling ended, spm status: SPM >2014-08-06 13:55:05,400 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-50) START, HSMClearTaskVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, taskId=9047e6c8-834e-4666-80c1-6158df8b31df), log id: 5b99dc3b >2014-08-06 13:55:05,446 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-50) FINISH, HSMClearTaskVDSCommand, log id: 5b99dc3b >2014-08-06 13:55:05,447 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-50) FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@472edc0c, log id: 4f376c77 >2014-08-06 13:55:05,451 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-50) Initialize Irs proxy from vds: 10.34.62.208 >2014-08-06 13:55:05,457 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-50) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208). >2014-08-06 13:55:05,467 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-30) START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 2b89ae74 >2014-08-06 13:55:05,497 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-30) -- executeIrsBrokerCommand: Attempting on storage pool 00000002-0002-0002-0002-0000000001d9 >2014-08-06 13:55:05,500 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-30) START, HSMGetAllTasksInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 27789e79 >2014-08-06 13:55:05,508 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-30) FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 27789e79 >2014-08-06 13:55:05,509 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-30) FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 2b89ae74 >2014-08-06 13:55:05,511 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (org.ovirt.thread.pool-8-thread-30) Discovered no tasks on Storage Pool Default >2014-08-06 14:30:54,837 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-47) Principal admin::internal synchronized >2014-08-06 14:30:55,356 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-49) Attempting to update VMs/Templates Ovf. >2014-08-06 15:30:54,844 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-5) Principal admin::internal synchronized >2014-08-06 15:30:55,374 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-18) Attempting to update VMs/Templates Ovf. >2014-08-06 16:30:54,851 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-63) Principal admin::internal synchronized >2014-08-06 16:30:55,403 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-84) Attempting to update VMs/Templates Ovf. >2014-08-06 16:54:53,193 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-43) domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 in problem. vds: rack >2014-08-06 16:54:53,197 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-43) domain 44559ec0-6bef-491c-bbd6-e67278f37987:export in problem. vds: rack >2014-08-06 16:56:25,530 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-79) Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2014-08-06 16:56:25,535 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-79) [6171df94] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-08-06 16:56:25,540 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-79) [6171df94] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Active to Unknown, reason : null >2014-08-06 16:56:25,542 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-79) [6171df94] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Active to Unknown, reason : null >2014-08-06 16:56:25,557 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-79) [6171df94] Correlation ID: 6171df94, Call Stack: null, Custom Event ID: -1, Message: Invalid status on Data Center Default. Setting Data Center status to Non Responsive (On host rack, Error: Cannot access master storage domain). >2014-08-06 16:58:25,582 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-79) [6171df94] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2014-08-06 16:58:25,585 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-79) [6171df94] hostFromVds::selectedVds - rack, spmStatus returned null! >2014-08-06 16:58:25,586 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-79) [6171df94] IrsBroker::Failed::GetStoragePoolInfoVDS >2014-08-06 16:58:25,586 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-79) [6171df94] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2014-08-06 16:58:25,625 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-26) [59cec128] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage >2014-08-06 16:58:25,649 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-26) [59cec128] START, ResetIrsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, ignoreStopFailed = true), log id: 61ffa4ff >2014-08-06 16:58:25,654 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-26) [59cec128] START, SpmStopVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9), log id: 2aea7969 >2014-08-06 16:58:25,660 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-26) [59cec128] SpmStopVDSCommand::Stopping SPM on vds rack, pool id 00000002-0002-0002-0002-0000000001d9 >2014-08-06 17:00:00,006 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-26) Autorecovering 1 storage domains >2014-08-06 17:00:00,008 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-26) Autorecovering storage domains id: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 >2014-08-06 17:00:00,016 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-26) [724f655c] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage >2014-08-06 17:00:00,020 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-26) [724f655c] ConnectDomainToStorage. Before Connect all hosts to pool. Time:8/6/14 5:00 PM >2014-08-06 17:00:00,100 INFO [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-8-thread-4) [7d4d024] Running command: ConnectStorageToVdsCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2014-08-06 17:00:00,106 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-4) [7d4d024] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 287108d7 >2014-08-06 17:00:25,557 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-60) Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2014-08-06 17:00:25,558 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-60) hostFromVds::selectedVds - rack, spmStatus returned null! >2014-08-06 17:00:25,559 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-60) IrsBroker::Failed::GetStoragePoolInfoVDS >2014-08-06 17:00:25,560 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-60) Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2014-08-06 17:01:25,560 WARN [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-26) [59cec128] could not stop spm of pool 00000002-0002-0002-0002-0000000001d9 on vds 17f7fe56-5f32-40ef-b098-611b84484e0b - reason: org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcRunTimeException: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException >2014-08-06 17:01:25,563 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-26) [59cec128] FINISH, SpmStopVDSCommand, log id: 2aea7969 >2014-08-06 17:01:25,569 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetStatsVDSCommand] (DefaultQuartzScheduler_Worker-93) Command GetStatsVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-08-06 17:01:25,566 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-4) [7d4d024] Command ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]) execution failed. Exception: VDSNetworkException: org.apache.commons.httpclient.NoHttpResponseException: The server 10.34.62.208 failed to respond >2014-08-06 17:01:25,577 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-13) Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-08-06 17:01:25,579 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-13) hostFromVds::selectedVds - rack, spmStatus returned null! >2014-08-06 17:01:25,577 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-4) [7d4d024] FINISH, ConnectStorageServerVDSCommand, log id: 287108d7 >2014-08-06 17:01:25,583 ERROR [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-8-thread-4) [7d4d024] Command org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: org.apache.commons.httpclient.NoHttpResponseException: The server 10.34.62.208 failed to respond (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-08-06 17:01:25,582 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-26) [59cec128] FINISH, ResetIrsVDSCommand, log id: 61ffa4ff >2014-08-06 17:01:25,596 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-93) Host rack is not responding. It will stay in Connecting state for a grace period of $80 seconds and after that an attempt to fence the host will be issued. >2014-08-06 17:01:25,597 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-93) vds::refreshVdsStats Failed getVdsStats, vds = 17f7fe56-5f32-40ef-b098-611b84484e0b : rack, error = VDSNetworkException: java.net.ConnectException: Connection refused >2014-08-06 17:01:25,605 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-26) [59cec128] Correlation ID: 59cec128, Job ID: 29d836ca-f288-40b0-a86d-3569b69bacf1, Call Stack: null, Custom Event ID: -1, Message: Failed to Reconstruct Master Domain for Data Center Default. >2014-08-06 17:01:25,609 ERROR [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-8-thread-4) [7d4d024] Transaction rolled-back for command: org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand. >2014-08-06 17:01:25,611 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-26) [724f655c] ConnectDomainToStorage. After Connect all hosts to pool. Time:8/6/14 5:01 PM >2014-08-06 17:01:25,605 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-93) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsStats(VdsUpdateRunTimeInfo.java:669) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:623) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor17.invoke(Unknown Source) [:1.7.0_55] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_55] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_55] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_55] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_55] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_55] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_55] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_55] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.getVdsStats(Unknown Source) > at sun.reflect.GeneratedMethodAccessor89.invoke(Unknown Source) [:1.7.0_55] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_55] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_55] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_55] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_55] > >2014-08-06 17:01:25,616 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-11) [6171df94] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue >2014-08-06 17:01:25,637 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-38) starting processDomainRecovery for domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 >2014-08-06 17:01:25,645 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-93) Failed to refresh VDS , vds = 17f7fe56-5f32-40ef-b098-611b84484e0b : rack, VDS Network Error, continuing. >java.net.ConnectException: Connection refused >2014-08-06 17:01:25,704 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-38) Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 was reported by all hosts in status UP as problematic. Not moving the domain to NonOperational because it is being reconstructed now. >2014-08-06 17:01:25,724 WARN [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-38) [401e1c49] CanDoAction of action ReconstructMasterDomain failed. Reasons:VAR__ACTION__RECONSTRUCT_MASTER,VAR__TYPE__STORAGE__DOMAIN,ACTION_TYPE_FAILED_NO_VDS_IN_POOL >2014-08-06 17:01:25,733 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-38) [401e1c49] Removing vds [] from the domain in maintenance cache >2014-08-06 17:01:25,734 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-38) [401e1c49] Removing host(s) [] from hosts unseen domain report cache >2014-08-06 17:01:25,735 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-11) [6171df94] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue >2014-08-06 17:01:25,737 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-30) starting processDomainRecovery for domain 44559ec0-6bef-491c-bbd6-e67278f37987:export >2014-08-06 17:01:25,743 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-30) Domain 44559ec0-6bef-491c-bbd6-e67278f37987:export was reported by all hosts in status UP as problematic. Moving the domain to NonOperational. >2014-08-06 17:01:25,760 INFO [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-30) [3378f4bb] Lock Acquired to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE >, sharedLocks= ] >2014-08-06 17:01:25,769 INFO [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-30) [3378f4bb] Running command: DeactivateStorageDomainCommand internal: true. Entities affected : ID: 44559ec0-6bef-491c-bbd6-e67278f37987 Type: Storage >2014-08-06 17:01:25,776 INFO [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-30) [3378f4bb] Lock freed to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE >, sharedLocks= ] >2014-08-06 17:01:25,787 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-30) [3378f4bb] Correlation ID: 3378f4bb, Job ID: 250ea4f2-24ce-4b5e-8fdb-00f7474e11b4, Call Stack: null, Custom Event ID: -1, Message: Storage Domain export (Data Center Default) was deactivated by system because it's not visible by any of the hosts. >2014-08-06 17:01:25,792 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-30) [3378f4bb] Removing vds [] from the domain in maintenance cache >2014-08-06 17:01:25,793 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-30) [3378f4bb] Removing host(s) [] from hosts unseen domain report cache >2014-08-06 17:01:28,927 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-39) START, GetHardwareInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]), log id: 3d95fa3 >2014-08-06 17:01:28,980 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-39) FINISH, GetHardwareInfoVDSCommand, log id: 3d95fa3 >2014-08-06 17:01:29,007 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-39) [557965d4] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS >2014-08-06 17:01:29,060 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-39) [1b1209d3] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-08-06 17:01:29,070 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-39) [18b3d48e] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-08-06 17:01:29,082 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-39) [18b3d48e] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, storageType = NFS, connectionList = [{ id: 05047c9a-152e-4ef2-b118-4a1753c1eae0, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35export, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 7b31f43f >2014-08-06 17:01:29,145 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-39) [18b3d48e] FINISH, ConnectStorageServerVDSCommand, return: {5083f370-af09-457b-9b73-3440e8b65a07=0, 05047c9a-152e-4ef2-b118-4a1753c1eae0=0}, log id: 7b31f43f >2014-08-06 17:01:29,146 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-39) [18b3d48e] Host rack storage connection was succeeded >2014-08-06 17:01:29,154 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-18) START, ConnectStoragePoolVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, masterVersion = 1), log id: 47a5a658 >2014-08-06 17:01:35,586 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-40) Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Inactive to Unknown, reason : no reporting hosts >2014-08-06 17:01:35,589 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-40) Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Inactive to Unknown, reason : no reporting hosts >2014-08-06 17:02:29,573 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-18) FINISH, ConnectStoragePoolVDSCommand, log id: 47a5a658 >2014-08-06 17:02:29,585 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-39) [18b3d48e] START, SetMOMPolicyParametersVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 18c000f0 >2014-08-06 17:02:29,678 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-39) [18b3d48e] FINISH, SetMOMPolicyParametersVDSCommand, log id: 18c000f0 >2014-08-06 17:02:29,686 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-39) [18b3d48e] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host rack was set to Up. >2014-08-06 17:02:29,694 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-39) [18b3d48e] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host rack. >2014-08-06 17:02:29,712 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-39) [587e61f3] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS >2014-08-06 17:02:35,708 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-56) hostFromVds::selectedVds - rack, spmStatus Free, storage pool Default >2014-08-06 17:02:35,719 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-56) starting spm on vds rack, storage pool Default, prevId -1, LVER -1 >2014-08-06 17:02:35,721 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-56) START, SpmStartVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, prevId=-1, prevLVER=-1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 778bc1f5 >2014-08-06 17:02:35,769 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-56) spmStart polling started: taskId = f716ddd6-20e5-44f2-8bb3-fcf92819a37e >2014-08-06 17:02:36,779 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-56) spmStart polling ended: taskId = f716ddd6-20e5-44f2-8bb3-fcf92819a37e task status = finished >2014-08-06 17:02:36,842 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-56) spmStart polling ended, spm status: SPM >2014-08-06 17:02:36,844 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-56) START, HSMClearTaskVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, taskId=f716ddd6-20e5-44f2-8bb3-fcf92819a37e), log id: 5c002b4 >2014-08-06 17:02:36,892 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-56) FINISH, HSMClearTaskVDSCommand, log id: 5c002b4 >2014-08-06 17:02:36,893 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-56) FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@4f44897, log id: 778bc1f5 >2014-08-06 17:02:36,899 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-56) Initialize Irs proxy from vds: 10.34.62.208 >2014-08-06 17:02:36,905 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-56) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208). >2014-08-06 17:02:36,916 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-9) START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 50716646 >2014-08-06 17:02:36,945 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-9) -- executeIrsBrokerCommand: Attempting on storage pool 00000002-0002-0002-0002-0000000001d9 >2014-08-06 17:02:36,947 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-9) START, HSMGetAllTasksInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 146488c4 >2014-08-06 17:02:36,955 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-9) FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 146488c4 >2014-08-06 17:02:36,956 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-9) FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 50716646 >2014-08-06 17:02:36,957 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (org.ovirt.thread.pool-8-thread-9) Discovered no tasks on Storage Pool Default >2014-08-06 17:30:54,859 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-35) Principal admin::internal synchronized >2014-08-06 17:30:55,421 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-50) Attempting to update VMs/Templates Ovf. >2014-08-06 18:30:54,867 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-26) [724f655c] Principal admin::internal synchronized >2014-08-06 18:30:55,457 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-24) Attempting to update VMs/Templates Ovf. >2014-08-06 19:30:54,874 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-87) Principal admin::internal synchronized >2014-08-06 19:30:55,470 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-91) Attempting to update VMs/Templates Ovf. >2014-08-06 20:02:26,159 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-16) domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 in problem. vds: rack >2014-08-06 20:02:26,162 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-16) domain 44559ec0-6bef-491c-bbd6-e67278f37987:export in problem. vds: rack >2014-08-06 20:03:53,279 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-60) Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2014-08-06 20:03:53,284 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-60) [6bd72a89] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-08-06 20:03:53,287 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-60) [6bd72a89] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Active to Unknown, reason : null >2014-08-06 20:03:53,289 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-60) [6bd72a89] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Active to Unknown, reason : null >2014-08-06 20:03:53,306 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-60) [6bd72a89] Correlation ID: 6bd72a89, Call Stack: null, Custom Event ID: -1, Message: Invalid status on Data Center Default. Setting Data Center status to Non Responsive (On host rack, Error: Cannot access master storage domain). >2014-08-06 20:05:53,331 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-60) [6bd72a89] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2014-08-06 20:05:53,334 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-60) [6bd72a89] hostFromVds::selectedVds - rack, spmStatus returned null! >2014-08-06 20:05:53,335 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-60) [6bd72a89] IrsBroker::Failed::GetStoragePoolInfoVDS >2014-08-06 20:05:53,336 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-60) [6bd72a89] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2014-08-06 20:05:53,372 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-25) [1f9ab2c5] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage >2014-08-06 20:05:53,390 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-25) [1f9ab2c5] START, ResetIrsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, ignoreStopFailed = true), log id: fa726cd >2014-08-06 20:05:53,394 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-25) [1f9ab2c5] START, SpmStopVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9), log id: 59322024 >2014-08-06 20:05:53,401 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-25) [1f9ab2c5] SpmStopVDSCommand::Stopping SPM on vds rack, pool id 00000002-0002-0002-0002-0000000001d9 >2014-08-06 20:07:53,306 WARN [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-25) [1f9ab2c5] could not stop spm of pool 00000002-0002-0002-0002-0000000001d9 on vds 17f7fe56-5f32-40ef-b098-611b84484e0b - reason: org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcRunTimeException: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException >2014-08-06 20:07:53,308 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-25) [1f9ab2c5] FINISH, SpmStopVDSCommand, log id: 59322024 >2014-08-06 20:07:53,310 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ListVDSCommand] (DefaultQuartzScheduler_Worker-73) Command ListVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-08-06 20:07:53,312 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-73) Host rack is not responding. It will stay in Connecting state for a grace period of $80 seconds and after that an attempt to fence the host will be issued. >2014-08-06 20:07:53,313 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-71) Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: VDSNetworkException: org.apache.commons.httpclient.NoHttpResponseException: The server 10.34.62.208 failed to respond >2014-08-06 20:07:53,315 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-25) [1f9ab2c5] FINISH, ResetIrsVDSCommand, log id: fa726cd >2014-08-06 20:07:53,316 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-73) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:407) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.fetchRunningVms(VdsUpdateRunTimeInfo.java:971) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVmStats(VdsUpdateRunTimeInfo.java:920) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:638) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor17.invoke(Unknown Source) [:1.7.0_55] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_55] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_55] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_55] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_55] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_55] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_55] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_55] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.list(Unknown Source) > at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source) [:1.7.0_55] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_55] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_55] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_55] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_55] > >2014-08-06 20:07:53,316 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-71) hostFromVds::selectedVds - rack, spmStatus returned null! >2014-08-06 20:07:53,333 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-25) [1f9ab2c5] Correlation ID: 1f9ab2c5, Job ID: 2149a573-d6d9-453e-889b-bc1e6b21ae08, Call Stack: null, Custom Event ID: -1, Message: Failed to Reconstruct Master Domain for Data Center Default. >2014-08-06 20:07:53,347 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-73) Failed to refresh VDS , vds = 17f7fe56-5f32-40ef-b098-611b84484e0b : rack, VDS Network Error, continuing. >java.net.ConnectException: Connection refused >2014-08-06 20:07:53,349 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-47) [6bd72a89] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue >2014-08-06 20:07:53,351 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-38) starting processDomainRecovery for domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 >2014-08-06 20:07:53,417 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-38) Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 was reported by all hosts in status UP as problematic. Not moving the domain to NonOperational because it is being reconstructed now. >2014-08-06 20:07:53,436 WARN [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-38) [253ddeb8] CanDoAction of action ReconstructMasterDomain failed. Reasons:VAR__ACTION__RECONSTRUCT_MASTER,VAR__TYPE__STORAGE__DOMAIN,ACTION_TYPE_FAILED_NO_VDS_IN_POOL >2014-08-06 20:07:53,443 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-38) [253ddeb8] Removing vds [] from the domain in maintenance cache >2014-08-06 20:07:53,444 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-38) [253ddeb8] Removing host(s) [] from hosts unseen domain report cache >2014-08-06 20:07:53,445 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-47) [6bd72a89] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue >2014-08-06 20:07:53,447 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-46) starting processDomainRecovery for domain 44559ec0-6bef-491c-bbd6-e67278f37987:export >2014-08-06 20:07:53,452 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-46) Domain 44559ec0-6bef-491c-bbd6-e67278f37987:export was reported by all hosts in status UP as problematic. Moving the domain to NonOperational. >2014-08-06 20:07:53,469 INFO [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-46) [9899ff9] Lock Acquired to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE >, sharedLocks= ] >2014-08-06 20:07:53,477 INFO [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-46) [9899ff9] Running command: DeactivateStorageDomainCommand internal: true. Entities affected : ID: 44559ec0-6bef-491c-bbd6-e67278f37987 Type: Storage >2014-08-06 20:07:53,483 INFO [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-46) [9899ff9] Lock freed to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE >, sharedLocks= ] >2014-08-06 20:07:53,495 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-46) [9899ff9] Correlation ID: 9899ff9, Job ID: 86c0d1b2-8b9d-4351-b998-e267423d2e7a, Call Stack: null, Custom Event ID: -1, Message: Storage Domain export (Data Center Default) was deactivated by system because it's not visible by any of the hosts. >2014-08-06 20:07:53,501 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-46) [9899ff9] Removing vds [] from the domain in maintenance cache >2014-08-06 20:07:53,502 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-46) [9899ff9] Removing host(s) [] from hosts unseen domain report cache >2014-08-06 20:07:56,605 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-100) START, GetHardwareInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]), log id: 53980f7b >2014-08-06 20:07:56,656 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-100) FINISH, GetHardwareInfoVDSCommand, log id: 53980f7b >2014-08-06 20:07:56,689 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-100) [23e2752a] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS >2014-08-06 20:07:56,702 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-100) [376829de] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-08-06 20:07:56,716 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-100) [2958405c] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-08-06 20:07:56,730 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-100) [2958405c] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, storageType = NFS, connectionList = [{ id: 05047c9a-152e-4ef2-b118-4a1753c1eae0, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35export, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 6c7563f7 >2014-08-06 20:07:56,792 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-100) [2958405c] FINISH, ConnectStorageServerVDSCommand, return: {5083f370-af09-457b-9b73-3440e8b65a07=0, 05047c9a-152e-4ef2-b118-4a1753c1eae0=0}, log id: 6c7563f7 >2014-08-06 20:07:56,793 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-100) [2958405c] Host rack storage connection was succeeded >2014-08-06 20:07:56,802 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-19) START, ConnectStoragePoolVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, masterVersion = 1), log id: 4570b3b8 >2014-08-06 20:08:03,345 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-41) Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Inactive to Unknown, reason : no reporting hosts >2014-08-06 20:08:03,348 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-41) Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Inactive to Unknown, reason : no reporting hosts >2014-08-06 20:08:57,213 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-19) FINISH, ConnectStoragePoolVDSCommand, log id: 4570b3b8 >2014-08-06 20:08:57,225 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-100) [2958405c] START, SetMOMPolicyParametersVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 7a0f0404 >2014-08-06 20:08:57,318 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-100) [2958405c] FINISH, SetMOMPolicyParametersVDSCommand, log id: 7a0f0404 >2014-08-06 20:08:57,326 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-100) [2958405c] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host rack was set to Up. >2014-08-06 20:08:57,332 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-100) [2958405c] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host rack. >2014-08-06 20:08:57,349 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-100) [498bd9ac] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS >2014-08-06 20:09:03,468 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-55) hostFromVds::selectedVds - rack, spmStatus Free, storage pool Default >2014-08-06 20:09:03,473 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-55) starting spm on vds rack, storage pool Default, prevId -1, LVER -1 >2014-08-06 20:09:03,476 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-55) START, SpmStartVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, prevId=-1, prevLVER=-1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 57f5b162 >2014-08-06 20:09:03,524 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-55) spmStart polling started: taskId = 29c76088-e3cc-40cf-b745-f462e7f5c7bd >2014-08-06 20:09:04,530 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-55) spmStart polling ended: taskId = 29c76088-e3cc-40cf-b745-f462e7f5c7bd task status = finished >2014-08-06 20:09:04,589 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-55) spmStart polling ended, spm status: SPM >2014-08-06 20:09:04,591 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-55) START, HSMClearTaskVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, taskId=29c76088-e3cc-40cf-b745-f462e7f5c7bd), log id: 648e7d39 >2014-08-06 20:09:04,636 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-55) FINISH, HSMClearTaskVDSCommand, log id: 648e7d39 >2014-08-06 20:09:04,637 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-55) FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@4eab0323, log id: 57f5b162 >2014-08-06 20:09:04,641 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-55) Initialize Irs proxy from vds: 10.34.62.208 >2014-08-06 20:09:04,648 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-55) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208). >2014-08-06 20:09:04,651 WARN [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-8-thread-13) Executing a command: java.util.concurrent.FutureTask , but note that there are 0 tasks in the queue. >2014-08-06 20:09:04,661 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-13) START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 5b02294 >2014-08-06 20:09:04,690 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-13) -- executeIrsBrokerCommand: Attempting on storage pool 00000002-0002-0002-0002-0000000001d9 >2014-08-06 20:09:04,692 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-13) START, HSMGetAllTasksInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 6620e161 >2014-08-06 20:09:04,698 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-13) FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 6620e161 >2014-08-06 20:09:04,699 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-13) FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 5b02294 >2014-08-06 20:09:04,699 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (org.ovirt.thread.pool-8-thread-13) Discovered no tasks on Storage Pool Default >2014-08-06 20:30:54,881 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-64) Principal admin::internal synchronized >2014-08-06 20:30:55,492 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-63) Attempting to update VMs/Templates Ovf. >2014-08-06 21:29:22,049 WARN [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-8-thread-2) Executing a command: java.util.concurrent.FutureTask , but note that there are 1 tasks in the queue. >2014-08-06 21:30:54,887 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-38) Principal admin::internal synchronized >2014-08-06 21:30:55,520 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-72) Attempting to update VMs/Templates Ovf. >2014-08-06 22:30:54,893 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-16) Principal admin::internal synchronized >2014-08-06 22:30:55,537 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-18) Attempting to update VMs/Templates Ovf. >2014-08-06 23:08:46,527 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-30) domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 in problem. vds: rack >2014-08-06 23:08:46,529 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-30) domain 44559ec0-6bef-491c-bbd6-e67278f37987:export in problem. vds: rack >2014-08-06 23:10:19,588 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-57) Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2014-08-06 23:10:19,596 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-57) [699d5b35] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-08-06 23:10:19,600 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-57) [699d5b35] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Active to Unknown, reason : null >2014-08-06 23:10:19,602 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-57) [699d5b35] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Active to Unknown, reason : null >2014-08-06 23:10:19,621 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-57) [699d5b35] Correlation ID: 699d5b35, Call Stack: null, Custom Event ID: -1, Message: Invalid status on Data Center Default. Setting Data Center status to Non Responsive (On host rack, Error: Cannot access master storage domain). >2014-08-06 23:12:19,638 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-57) [699d5b35] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2014-08-06 23:12:19,640 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-57) [699d5b35] hostFromVds::selectedVds - rack, spmStatus returned null! >2014-08-06 23:12:19,641 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-57) [699d5b35] IrsBroker::Failed::GetStoragePoolInfoVDS >2014-08-06 23:12:19,642 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-57) [699d5b35] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2014-08-06 23:12:19,701 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-10) [12f9d30] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage >2014-08-06 23:12:19,728 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-10) [12f9d30] START, ResetIrsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, ignoreStopFailed = true), log id: 7bd8f602 >2014-08-06 23:12:19,732 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-10) [12f9d30] START, SpmStopVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9), log id: 1957f05e >2014-08-06 23:12:19,738 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-10) [12f9d30] SpmStopVDSCommand::Stopping SPM on vds rack, pool id 00000002-0002-0002-0002-0000000001d9 >2014-08-06 23:14:19,617 WARN [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-10) [12f9d30] could not stop spm of pool 00000002-0002-0002-0002-0000000001d9 on vds 17f7fe56-5f32-40ef-b098-611b84484e0b - reason: org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcRunTimeException: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException >2014-08-06 23:14:19,620 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-10) [12f9d30] FINISH, SpmStopVDSCommand, log id: 1957f05e >2014-08-06 23:14:19,621 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-32) Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: VDSNetworkException: org.apache.commons.httpclient.NoHttpResponseException: The server 10.34.62.208 failed to respond >2014-08-06 23:14:19,623 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-32) hostFromVds::selectedVds - rack, spmStatus returned null! >2014-08-06 23:14:19,620 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ListVDSCommand] (DefaultQuartzScheduler_Worker-43) Command ListVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-08-06 23:14:19,628 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-43) Host rack is not responding. It will stay in Connecting state for a grace period of $80 seconds and after that an attempt to fence the host will be issued. >2014-08-06 23:14:19,629 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-43) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:407) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.fetchRunningVms(VdsUpdateRunTimeInfo.java:971) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVmStats(VdsUpdateRunTimeInfo.java:920) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:638) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor17.invoke(Unknown Source) [:1.7.0_55] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_55] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_55] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_55] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_55] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_55] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_55] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_55] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.list(Unknown Source) > at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source) [:1.7.0_55] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_55] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_55] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_55] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_55] > >2014-08-06 23:14:19,627 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-10) [12f9d30] FINISH, ResetIrsVDSCommand, log id: 7bd8f602 >2014-08-06 23:14:19,663 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-10) [12f9d30] Correlation ID: 12f9d30, Job ID: 023e5721-b319-42d0-95fc-9e7016679777, Call Stack: null, Custom Event ID: -1, Message: Failed to Reconstruct Master Domain for Data Center Default. >2014-08-06 23:14:19,668 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-43) Failed to refresh VDS , vds = 17f7fe56-5f32-40ef-b098-611b84484e0b : rack, VDS Network Error, continuing. >java.net.ConnectException: Connection refused >2014-08-06 23:14:19,674 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-1) [699d5b35] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue >2014-08-06 23:14:19,676 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-19) starting processDomainRecovery for domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 >2014-08-06 23:14:19,739 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-19) Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 was reported by all hosts in status UP as problematic. Not moving the domain to NonOperational because it is being reconstructed now. >2014-08-06 23:14:19,760 WARN [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-19) [3cc3fb88] CanDoAction of action ReconstructMasterDomain failed. Reasons:VAR__ACTION__RECONSTRUCT_MASTER,VAR__TYPE__STORAGE__DOMAIN,ACTION_TYPE_FAILED_NO_VDS_IN_POOL >2014-08-06 23:14:19,768 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-19) [3cc3fb88] Removing vds [] from the domain in maintenance cache >2014-08-06 23:14:19,769 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-19) [3cc3fb88] Removing host(s) [] from hosts unseen domain report cache >2014-08-06 23:14:19,769 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-1) [699d5b35] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue >2014-08-06 23:14:19,771 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-17) starting processDomainRecovery for domain 44559ec0-6bef-491c-bbd6-e67278f37987:export >2014-08-06 23:14:19,776 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-17) Domain 44559ec0-6bef-491c-bbd6-e67278f37987:export was reported by all hosts in status UP as problematic. Moving the domain to NonOperational. >2014-08-06 23:14:19,792 INFO [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-17) [3893d5cd] Lock Acquired to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE >, sharedLocks= ] >2014-08-06 23:14:19,798 INFO [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-17) [3893d5cd] Running command: DeactivateStorageDomainCommand internal: true. Entities affected : ID: 44559ec0-6bef-491c-bbd6-e67278f37987 Type: Storage >2014-08-06 23:14:19,805 INFO [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-17) [3893d5cd] Lock freed to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE >, sharedLocks= ] >2014-08-06 23:14:19,827 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-17) [3893d5cd] Correlation ID: 3893d5cd, Job ID: 1f799972-e1d4-4606-b535-3f01510c18ff, Call Stack: null, Custom Event ID: -1, Message: Storage Domain export (Data Center Default) was deactivated by system because it's not visible by any of the hosts. >2014-08-06 23:14:19,832 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-17) [3893d5cd] Removing vds [] from the domain in maintenance cache >2014-08-06 23:14:19,833 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-17) [3893d5cd] Removing host(s) [] from hosts unseen domain report cache >2014-08-06 23:14:22,928 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-81) START, GetHardwareInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]), log id: 230a6365 >2014-08-06 23:14:22,977 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-81) FINISH, GetHardwareInfoVDSCommand, log id: 230a6365 >2014-08-06 23:14:23,011 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-81) [3dc0f67d] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS >2014-08-06 23:14:23,025 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-81) [10eae19] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-08-06 23:14:23,037 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-81) [327de80e] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-08-06 23:14:23,047 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-81) [327de80e] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, storageType = NFS, connectionList = [{ id: 05047c9a-152e-4ef2-b118-4a1753c1eae0, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35export, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 4973249 >2014-08-06 23:14:23,109 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-81) [327de80e] FINISH, ConnectStorageServerVDSCommand, return: {5083f370-af09-457b-9b73-3440e8b65a07=0, 05047c9a-152e-4ef2-b118-4a1753c1eae0=0}, log id: 4973249 >2014-08-06 23:14:23,110 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-81) [327de80e] Host rack storage connection was succeeded >2014-08-06 23:14:23,119 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-28) START, ConnectStoragePoolVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, masterVersion = 1), log id: 23a021b3 >2014-08-06 23:14:29,631 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-17) Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Inactive to Unknown, reason : no reporting hosts >2014-08-06 23:14:29,634 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-17) Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Inactive to Unknown, reason : no reporting hosts >2014-08-06 23:15:23,524 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-28) FINISH, ConnectStoragePoolVDSCommand, log id: 23a021b3 >2014-08-06 23:15:23,538 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-81) [327de80e] START, SetMOMPolicyParametersVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 303fd306 >2014-08-06 23:15:23,633 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-81) [327de80e] FINISH, SetMOMPolicyParametersVDSCommand, log id: 303fd306 >2014-08-06 23:15:23,643 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-81) [327de80e] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host rack was set to Up. >2014-08-06 23:15:23,651 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-81) [327de80e] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host rack. >2014-08-06 23:15:23,677 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-81) [57c3242a] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS >2014-08-06 23:15:29,786 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-58) [ba1ce2a] hostFromVds::selectedVds - rack, spmStatus Free, storage pool Default >2014-08-06 23:15:29,791 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-58) [ba1ce2a] starting spm on vds rack, storage pool Default, prevId -1, LVER -1 >2014-08-06 23:15:29,793 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-58) [ba1ce2a] START, SpmStartVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, prevId=-1, prevLVER=-1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 167050c4 >2014-08-06 23:15:29,842 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-58) [ba1ce2a] spmStart polling started: taskId = f088c5f5-7c5a-409b-b852-c2f3a3f27f4f >2014-08-06 23:15:30,849 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-58) [ba1ce2a] spmStart polling ended: taskId = f088c5f5-7c5a-409b-b852-c2f3a3f27f4f task status = finished >2014-08-06 23:15:30,906 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-58) [ba1ce2a] spmStart polling ended, spm status: SPM >2014-08-06 23:15:30,908 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-58) [ba1ce2a] START, HSMClearTaskVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, taskId=f088c5f5-7c5a-409b-b852-c2f3a3f27f4f), log id: f833712 >2014-08-06 23:15:30,953 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-58) [ba1ce2a] FINISH, HSMClearTaskVDSCommand, log id: f833712 >2014-08-06 23:15:30,954 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-58) [ba1ce2a] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@3cf346ed, log id: 167050c4 >2014-08-06 23:15:30,958 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-58) [ba1ce2a] Initialize Irs proxy from vds: 10.34.62.208 >2014-08-06 23:15:30,965 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-58) [ba1ce2a] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208). >2014-08-06 23:15:30,966 WARN [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-8-thread-40) Executing a command: java.util.concurrent.FutureTask , but note that there are 1 tasks in the queue. >2014-08-06 23:15:30,981 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-40) [ba1ce2a] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 376e2e1e >2014-08-06 23:15:31,015 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-40) [ba1ce2a] -- executeIrsBrokerCommand: Attempting on storage pool 00000002-0002-0002-0002-0000000001d9 >2014-08-06 23:15:31,016 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-40) [ba1ce2a] START, HSMGetAllTasksInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 23a630b6 >2014-08-06 23:15:31,022 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-40) [ba1ce2a] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 23a630b6 >2014-08-06 23:15:31,022 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-40) [ba1ce2a] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 376e2e1e >2014-08-06 23:15:31,023 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (org.ovirt.thread.pool-8-thread-40) [ba1ce2a] Discovered no tasks on Storage Pool Default >2014-08-06 23:30:54,903 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-20) Principal admin::internal synchronized >2014-08-06 23:30:55,550 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-93) Attempting to update VMs/Templates Ovf. >2014-08-07 00:30:54,915 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-63) Principal admin::internal synchronized >2014-08-07 00:30:55,583 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-43) Attempting to update VMs/Templates Ovf. >2014-08-07 01:30:54,921 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-45) Principal admin::internal synchronized >2014-08-07 01:30:55,612 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-55) Attempting to update VMs/Templates Ovf. >2014-08-07 02:00:41,521 WARN [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-8-thread-35) Executing a command: java.util.concurrent.FutureTask , but note that there are 1 tasks in the queue. >2014-08-07 02:15:21,896 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-9) domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 in problem. vds: rack >2014-08-07 02:15:21,900 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-9) domain 44559ec0-6bef-491c-bbd6-e67278f37987:export in problem. vds: rack >2014-08-07 02:16:47,579 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-14) [6bdb6450] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2014-08-07 02:16:47,584 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-14) [2bb2f45d] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-08-07 02:16:47,590 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-14) [2bb2f45d] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Active to Unknown, reason : null >2014-08-07 02:16:47,592 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-14) [2bb2f45d] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Active to Unknown, reason : null >2014-08-07 02:16:47,633 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-14) [2bb2f45d] Correlation ID: 2bb2f45d, Call Stack: null, Custom Event ID: -1, Message: Invalid status on Data Center Default. Setting Data Center status to Non Responsive (On host rack, Error: Cannot access master storage domain). >2014-08-07 02:18:47,634 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-14) [2bb2f45d] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2014-08-07 02:18:47,637 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-14) [2bb2f45d] hostFromVds::selectedVds - rack, spmStatus returned null! >2014-08-07 02:18:47,638 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-14) [2bb2f45d] IrsBroker::Failed::GetStoragePoolInfoVDS >2014-08-07 02:18:47,639 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-14) [2bb2f45d] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2014-08-07 02:18:47,680 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-29) [64444835] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage >2014-08-07 02:18:47,707 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-29) [64444835] START, ResetIrsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, ignoreStopFailed = true), log id: 7c3c3233 >2014-08-07 02:18:47,711 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-29) [64444835] START, SpmStopVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9), log id: 7944a00a >2014-08-07 02:18:47,716 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-29) [64444835] SpmStopVDSCommand::Stopping SPM on vds rack, pool id 00000002-0002-0002-0002-0000000001d9 >2014-08-07 02:20:00,005 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-65) Autorecovering 1 storage domains >2014-08-07 02:20:00,007 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-65) Autorecovering storage domains id: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 >2014-08-07 02:20:00,009 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-65) [126267c5] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage >2014-08-07 02:20:00,011 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-65) [126267c5] ConnectDomainToStorage. Before Connect all hosts to pool. Time:8/7/14 2:20 AM >2014-08-07 02:20:00,022 INFO [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-8-thread-13) [4563d8c4] Running command: ConnectStorageToVdsCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2014-08-07 02:20:00,027 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-13) [4563d8c4] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 5061cfd3 >2014-08-07 02:20:47,606 WARN [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-29) [64444835] could not stop spm of pool 00000002-0002-0002-0002-0000000001d9 on vds 17f7fe56-5f32-40ef-b098-611b84484e0b - reason: org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcRunTimeException: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException >2014-08-07 02:20:47,608 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-29) [64444835] FINISH, SpmStopVDSCommand, log id: 7944a00a >2014-08-07 02:20:47,613 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-36) Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: VDSNetworkException: org.apache.commons.httpclient.NoHttpResponseException: The server 10.34.62.208 failed to respond >2014-08-07 02:20:47,618 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-36) hostFromVds::selectedVds - rack, spmStatus returned null! >2014-08-07 02:20:47,616 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ListVDSCommand] (DefaultQuartzScheduler_Worker-60) [6bd72a89] Command ListVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-08-07 02:20:47,620 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-60) [6bd72a89] Host rack is not responding. It will stay in Connecting state for a grace period of $80 seconds and after that an attempt to fence the host will be issued. >2014-08-07 02:20:47,622 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-60) [6bd72a89] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:407) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.fetchRunningVms(VdsUpdateRunTimeInfo.java:971) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVmStats(VdsUpdateRunTimeInfo.java:920) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:638) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor17.invoke(Unknown Source) [:1.7.0_55] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_55] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_55] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_55] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_55] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_55] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_55] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_55] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.list(Unknown Source) > at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source) [:1.7.0_55] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_55] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_55] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_55] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_55] > >2014-08-07 02:20:47,617 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-13) [4563d8c4] Command ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-08-07 02:20:47,626 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-29) [64444835] FINISH, ResetIrsVDSCommand, log id: 7c3c3233 >2014-08-07 02:20:47,647 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-13) [4563d8c4] FINISH, ConnectStorageServerVDSCommand, log id: 5061cfd3 >2014-08-07 02:20:47,649 ERROR [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-8-thread-13) [4563d8c4] Command org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-08-07 02:20:47,674 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-29) [64444835] Correlation ID: 64444835, Job ID: 4eea4f15-b99f-4535-94bc-03e53ed61c70, Call Stack: null, Custom Event ID: -1, Message: Failed to Reconstruct Master Domain for Data Center Default. >2014-08-07 02:20:47,679 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-50) [2bb2f45d] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue >2014-08-07 02:20:47,681 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-25) starting processDomainRecovery for domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 >2014-08-07 02:20:47,691 ERROR [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-8-thread-13) [4563d8c4] Transaction rolled-back for command: org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand. >2014-08-07 02:20:47,692 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-65) [126267c5] ConnectDomainToStorage. After Connect all hosts to pool. Time:8/7/14 2:20 AM >2014-08-07 02:20:47,696 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-60) [6bd72a89] Failed to refresh VDS , vds = 17f7fe56-5f32-40ef-b098-611b84484e0b : rack, VDS Network Error, continuing. >java.net.ConnectException: Connection refused >2014-08-07 02:20:47,767 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-25) Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 was reported by all hosts in status UP as problematic. Not moving the domain to NonOperational because it is being reconstructed now. >2014-08-07 02:20:47,786 WARN [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-25) [1b36501d] CanDoAction of action ReconstructMasterDomain failed. Reasons:VAR__ACTION__RECONSTRUCT_MASTER,VAR__TYPE__STORAGE__DOMAIN,ACTION_TYPE_FAILED_NO_VDS_IN_POOL >2014-08-07 02:20:47,794 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-25) [1b36501d] Removing vds [] from the domain in maintenance cache >2014-08-07 02:20:47,795 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-25) [1b36501d] Removing host(s) [] from hosts unseen domain report cache >2014-08-07 02:20:47,796 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-50) [2bb2f45d] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue >2014-08-07 02:20:47,798 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-22) starting processDomainRecovery for domain 44559ec0-6bef-491c-bbd6-e67278f37987:export >2014-08-07 02:20:47,804 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-22) Domain 44559ec0-6bef-491c-bbd6-e67278f37987:export was reported by all hosts in status UP as problematic. Moving the domain to NonOperational. >2014-08-07 02:20:47,821 INFO [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-22) [6c88dc22] Lock Acquired to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE >, sharedLocks= ] >2014-08-07 02:20:47,828 INFO [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-22) [6c88dc22] Running command: DeactivateStorageDomainCommand internal: true. Entities affected : ID: 44559ec0-6bef-491c-bbd6-e67278f37987 Type: Storage >2014-08-07 02:20:47,833 INFO [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-22) [6c88dc22] Lock freed to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE >, sharedLocks= ] >2014-08-07 02:20:47,843 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-22) [6c88dc22] Correlation ID: 6c88dc22, Job ID: 6cd6d2cb-3320-4e6a-a07a-be504492be85, Call Stack: null, Custom Event ID: -1, Message: Storage Domain export (Data Center Default) was deactivated by system because it's not visible by any of the hosts. >2014-08-07 02:20:47,848 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-22) [6c88dc22] Removing vds [] from the domain in maintenance cache >2014-08-07 02:20:47,849 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-22) [6c88dc22] Removing host(s) [] from hosts unseen domain report cache >2014-08-07 02:20:50,964 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-51) START, GetHardwareInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]), log id: d6d2d1c >2014-08-07 02:20:51,021 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-51) FINISH, GetHardwareInfoVDSCommand, log id: d6d2d1c >2014-08-07 02:20:51,052 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-51) [17cc77] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS >2014-08-07 02:20:51,104 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-51) [1d91d6a5] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-08-07 02:20:51,113 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-51) [515c423b] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-08-07 02:20:51,119 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-51) [515c423b] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, storageType = NFS, connectionList = [{ id: 05047c9a-152e-4ef2-b118-4a1753c1eae0, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35export, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 34583e2 >2014-08-07 02:20:51,183 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-51) [515c423b] FINISH, ConnectStorageServerVDSCommand, return: {5083f370-af09-457b-9b73-3440e8b65a07=0, 05047c9a-152e-4ef2-b118-4a1753c1eae0=0}, log id: 34583e2 >2014-08-07 02:20:51,184 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-51) [515c423b] Host rack storage connection was succeeded >2014-08-07 02:20:51,190 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-19) START, ConnectStoragePoolVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, masterVersion = 1), log id: 1b52c375 >2014-08-07 02:20:57,623 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-23) Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Inactive to Unknown, reason : no reporting hosts >2014-08-07 02:20:57,627 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-23) Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Inactive to Unknown, reason : no reporting hosts >2014-08-07 02:21:51,588 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-19) FINISH, ConnectStoragePoolVDSCommand, log id: 1b52c375 >2014-08-07 02:21:51,601 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-51) [515c423b] START, SetMOMPolicyParametersVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 3eab5140 >2014-08-07 02:21:51,700 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-51) [515c423b] FINISH, SetMOMPolicyParametersVDSCommand, log id: 3eab5140 >2014-08-07 02:21:51,709 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-51) [515c423b] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host rack was set to Up. >2014-08-07 02:21:51,715 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-51) [515c423b] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host rack. >2014-08-07 02:21:51,734 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-51) [5b995d2] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS >2014-08-07 02:21:57,740 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-18) hostFromVds::selectedVds - rack, spmStatus Free, storage pool Default >2014-08-07 02:21:57,745 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-18) starting spm on vds rack, storage pool Default, prevId -1, LVER -1 >2014-08-07 02:21:57,747 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-18) START, SpmStartVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, prevId=-1, prevLVER=-1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 5c540c37 >2014-08-07 02:21:57,818 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-18) spmStart polling started: taskId = 6571e5b5-148c-4dc4-849e-f0a184206957 >2014-08-07 02:21:58,825 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-18) spmStart polling ended: taskId = 6571e5b5-148c-4dc4-849e-f0a184206957 task status = finished >2014-08-07 02:21:58,881 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-18) spmStart polling ended, spm status: SPM >2014-08-07 02:21:58,883 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-18) START, HSMClearTaskVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, taskId=6571e5b5-148c-4dc4-849e-f0a184206957), log id: 5aa8a24c >2014-08-07 02:21:58,930 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-18) FINISH, HSMClearTaskVDSCommand, log id: 5aa8a24c >2014-08-07 02:21:58,931 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-18) FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@6896d515, log id: 5c540c37 >2014-08-07 02:21:58,936 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-18) Initialize Irs proxy from vds: 10.34.62.208 >2014-08-07 02:21:58,945 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-18) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208). >2014-08-07 02:21:58,953 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-38) START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: bd502cd >2014-08-07 02:21:58,990 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-38) -- executeIrsBrokerCommand: Attempting on storage pool 00000002-0002-0002-0002-0000000001d9 >2014-08-07 02:21:58,993 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-38) START, HSMGetAllTasksInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 1631d787 >2014-08-07 02:21:58,998 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-38) FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 1631d787 >2014-08-07 02:21:58,999 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-38) FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: bd502cd >2014-08-07 02:21:59,000 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (org.ovirt.thread.pool-8-thread-38) Discovered no tasks on Storage Pool Default >2014-08-07 02:30:54,926 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-61) Principal admin::internal synchronized >2014-08-07 02:30:55,626 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-59) Attempting to update VMs/Templates Ovf. >2014-08-07 03:30:54,937 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-10) Principal admin::internal synchronized >2014-08-07 03:30:55,643 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-27) Attempting to update VMs/Templates Ovf. >2014-08-07 03:35:35,001 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-93) Start deleteAgedOutAuditLogs >2014-08-07 03:35:35,001 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-17) Start deleteAgedOutCommandEntities >2014-08-07 03:35:35,033 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-93) Finished deleteAgedOutAuditLogs >2014-08-07 03:35:35,038 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-17) Finished deleteAgedOutCommandEntities >2014-08-07 04:30:54,944 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-85) Principal admin::internal synchronized >2014-08-07 04:30:55,676 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-86) Attempting to update VMs/Templates Ovf. >2014-08-07 05:21:41,345 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-40) domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 in problem. vds: rack >2014-08-07 05:21:41,348 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-40) domain 44559ec0-6bef-491c-bbd6-e67278f37987:export in problem. vds: rack >2014-08-07 05:23:14,254 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-34) Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2014-08-07 05:23:14,264 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-34) [6d1d4af0] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-08-07 05:23:14,274 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-34) [6d1d4af0] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Active to Unknown, reason : null >2014-08-07 05:23:14,277 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-34) [6d1d4af0] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Active to Unknown, reason : null >2014-08-07 05:23:14,292 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-34) [6d1d4af0] Correlation ID: 6d1d4af0, Call Stack: null, Custom Event ID: -1, Message: Invalid status on Data Center Default. Setting Data Center status to Non Responsive (On host rack, Error: Cannot access master storage domain). >2014-08-07 05:25:14,302 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-34) [6d1d4af0] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2014-08-07 05:25:14,305 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-34) [6d1d4af0] hostFromVds::selectedVds - rack, spmStatus returned null! >2014-08-07 05:25:14,307 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-34) [6d1d4af0] IrsBroker::Failed::GetStoragePoolInfoVDS >2014-08-07 05:25:14,308 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-34) [6d1d4af0] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2014-08-07 05:25:14,354 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-8) [41ce916] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage >2014-08-07 05:25:14,375 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-8) [41ce916] START, ResetIrsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, ignoreStopFailed = true), log id: 59b92999 >2014-08-07 05:25:14,379 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-8) [41ce916] START, SpmStopVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9), log id: 22a7ce4f >2014-08-07 05:25:14,384 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-8) [41ce916] SpmStopVDSCommand::Stopping SPM on vds rack, pool id 00000002-0002-0002-0002-0000000001d9 >2014-08-07 05:27:14,279 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-9) Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2014-08-07 05:27:14,281 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-9) hostFromVds::selectedVds - rack, spmStatus returned null! >2014-08-07 05:27:14,282 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-9) IrsBroker::Failed::GetStoragePoolInfoVDS >2014-08-07 05:27:14,283 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-9) Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2014-08-07 05:28:14,387 WARN [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-8) [41ce916] could not stop spm of pool 00000002-0002-0002-0002-0000000001d9 on vds 17f7fe56-5f32-40ef-b098-611b84484e0b - reason: org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcRunTimeException: java.util.concurrent.TimeoutException >2014-08-07 05:28:14,389 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-8) [41ce916] FINISH, SpmStopVDSCommand, log id: 22a7ce4f >2014-08-07 05:28:14,396 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-8) [41ce916] FINISH, ResetIrsVDSCommand, log id: 59b92999 >2014-08-07 05:28:14,423 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-8) [41ce916] Correlation ID: 41ce916, Job ID: a0dc1eea-9aba-4ce2-8479-af836043beee, Call Stack: null, Custom Event ID: -1, Message: Failed to Reconstruct Master Domain for Data Center Default. >2014-08-07 05:28:14,431 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-27) [6d1d4af0] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue >2014-08-07 05:28:14,433 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-7) starting processDomainRecovery for domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 >2014-08-07 05:28:14,531 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-7) Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 was reported by all hosts in status UP as problematic. Not moving the domain to NonOperational because it is being reconstructed now. >2014-08-07 05:28:14,564 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-7) [3143c641] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage >2014-08-07 05:28:14,598 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-7) [3143c641] Removing vds [] from the domain in maintenance cache >2014-08-07 05:28:14,599 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-7) [3143c641] Removing host(s) [] from hosts unseen domain report cache >2014-08-07 05:28:14,600 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-27) [6d1d4af0] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue >2014-08-07 05:28:14,608 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-28) starting processDomainRecovery for domain 44559ec0-6bef-491c-bbd6-e67278f37987:export >2014-08-07 05:28:14,614 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-28) Domain 44559ec0-6bef-491c-bbd6-e67278f37987:export was reported by all hosts in status UP as problematic. Moving the domain to NonOperational. >2014-08-07 05:28:14,631 INFO [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-28) [2ab57563] Lock Acquired to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE >, sharedLocks= ] >2014-08-07 05:28:14,638 INFO [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-28) [2ab57563] Running command: DeactivateStorageDomainCommand internal: true. Entities affected : ID: 44559ec0-6bef-491c-bbd6-e67278f37987 Type: Storage >2014-08-07 05:28:14,644 INFO [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-28) [2ab57563] Lock freed to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE >, sharedLocks= ] >2014-08-07 05:28:14,654 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-28) [2ab57563] Correlation ID: 2ab57563, Job ID: 868327ef-28fb-4a80-8fbc-6dc60a52f9f9, Call Stack: null, Custom Event ID: -1, Message: Storage Domain export (Data Center Default) was deactivated by system because it's not visible by any of the hosts. >2014-08-07 05:28:14,659 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-28) [2ab57563] Removing vds [] from the domain in maintenance cache >2014-08-07 05:28:14,660 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-28) [2ab57563] Removing host(s) [] from hosts unseen domain report cache >2014-08-07 05:29:14,290 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-62) Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2014-08-07 05:29:14,292 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-62) hostFromVds::selectedVds - rack, spmStatus returned null! >2014-08-07 05:29:14,292 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-62) IrsBroker::Failed::GetStoragePoolInfoVDS >2014-08-07 05:29:14,293 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-62) Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2014-08-07 05:29:14,329 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-3) [30318db] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage >2014-08-07 05:29:14,357 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-4) Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue >2014-08-07 05:30:00,009 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-100) [498bd9ac] Autorecovering 2 storage domains >2014-08-07 05:30:00,011 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-100) [498bd9ac] Autorecovering storage domains id: 44559ec0-6bef-491c-bbd6-e67278f37987 >2014-08-07 05:30:00,014 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-100) [14b3cca1] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 44559ec0-6bef-491c-bbd6-e67278f37987 Type: Storage >2014-08-07 05:30:00,016 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-100) [14b3cca1] ConnectDomainToStorage. Before Connect all hosts to pool. Time:8/7/14 5:30 AM >2014-08-07 05:30:00,029 INFO [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-8-thread-16) [8682d24] Running command: ConnectStorageToVdsCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2014-08-07 05:30:00,033 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-16) [8682d24] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: 05047c9a-152e-4ef2-b118-4a1753c1eae0, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35export, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 758a4431 >2014-08-07 05:30:54,952 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-72) Principal admin::internal synchronized >2014-08-07 05:30:55,691 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-61) Attempting to update VMs/Templates Ovf. >2014-08-07 05:31:00,048 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-16) [8682d24] FINISH, ConnectStorageServerVDSCommand, return: {05047c9a-152e-4ef2-b118-4a1753c1eae0=0}, log id: 758a4431 >2014-08-07 05:31:00,054 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-100) [14b3cca1] ConnectDomainToStorage. After Connect all hosts to pool. Time:8/7/14 5:31 AM >2014-08-07 05:31:00,056 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-100) [14b3cca1] Autorecovering storage domains id: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 >2014-08-07 05:31:00,058 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-100) [4c525b57] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage >2014-08-07 05:31:00,059 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-100) [4c525b57] ConnectDomainToStorage. Before Connect all hosts to pool. Time:8/7/14 5:31 AM >2014-08-07 05:31:00,068 INFO [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-8-thread-7) [7b7cc41d] Running command: ConnectStorageToVdsCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2014-08-07 05:31:00,072 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-7) [7b7cc41d] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 64e390b6 >2014-08-07 05:31:14,306 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ListVDSCommand] (DefaultQuartzScheduler_Worker-69) Command ListVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-08-07 05:31:14,307 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-69) Host rack is not responding. It will stay in Connecting state for a grace period of $60 seconds and after that an attempt to fence the host will be issued. >2014-08-07 05:31:14,308 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-69) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:407) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.fetchRunningVms(VdsUpdateRunTimeInfo.java:971) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVmStats(VdsUpdateRunTimeInfo.java:920) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:638) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor17.invoke(Unknown Source) [:1.7.0_55] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_55] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_55] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_55] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_55] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_55] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_55] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_55] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.list(Unknown Source) > at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source) [:1.7.0_55] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_55] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_55] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_55] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_55] > >2014-08-07 05:31:14,313 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-7) [7b7cc41d] Command ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]) execution failed. Exception: VDSNetworkException: org.apache.commons.httpclient.NoHttpResponseException: The server 10.34.62.208 failed to respond >2014-08-07 05:31:14,312 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-78) Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: VDSNetworkException: org.apache.commons.httpclient.NoHttpResponseException: The server 10.34.62.208 failed to respond >2014-08-07 05:31:14,354 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-78) hostFromVds::selectedVds - rack, spmStatus returned null! >2014-08-07 05:31:14,353 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-7) [7b7cc41d] FINISH, ConnectStorageServerVDSCommand, log id: 64e390b6 >2014-08-07 05:31:14,356 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-69) Failed to refresh VDS , vds = 17f7fe56-5f32-40ef-b098-611b84484e0b : rack, VDS Network Error, continuing. >java.net.ConnectException: Connection refused >2014-08-07 05:31:14,356 ERROR [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-8-thread-7) [7b7cc41d] Command org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: org.apache.commons.httpclient.NoHttpResponseException: The server 10.34.62.208 failed to respond (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-08-07 05:31:14,363 ERROR [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-8-thread-7) [7b7cc41d] Transaction rolled-back for command: org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand. >2014-08-07 05:31:14,364 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-100) [4c525b57] ConnectDomainToStorage. After Connect all hosts to pool. Time:8/7/14 5:31 AM >2014-08-07 05:31:17,632 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-94) START, GetHardwareInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]), log id: de53fef >2014-08-07 05:31:17,693 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-94) FINISH, GetHardwareInfoVDSCommand, log id: de53fef >2014-08-07 05:31:17,724 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-94) [375a3aa3] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS >2014-08-07 05:31:17,740 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-94) [188e29b7] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-08-07 05:31:17,749 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-94) [6e904378] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-08-07 05:31:17,763 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-94) [6e904378] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, storageType = NFS, connectionList = [{ id: 05047c9a-152e-4ef2-b118-4a1753c1eae0, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35export, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 7d6272c3 >2014-08-07 05:31:17,821 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-94) [6e904378] FINISH, ConnectStorageServerVDSCommand, return: {5083f370-af09-457b-9b73-3440e8b65a07=0, 05047c9a-152e-4ef2-b118-4a1753c1eae0=0}, log id: 7d6272c3 >2014-08-07 05:31:17,822 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-94) [6e904378] Host rack storage connection was succeeded >2014-08-07 05:31:17,831 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-25) START, ConnectStoragePoolVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, masterVersion = 1), log id: 43449517 >2014-08-07 05:31:24,359 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-5) Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Inactive to Unknown, reason : no reporting hosts >2014-08-07 05:31:24,362 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-5) Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Inactive to Unknown, reason : no reporting hosts >2014-08-07 05:32:18,227 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-25) FINISH, ConnectStoragePoolVDSCommand, log id: 43449517 >2014-08-07 05:32:18,241 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-94) [6e904378] START, SetMOMPolicyParametersVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 4161db52 >2014-08-07 05:32:18,333 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-94) [6e904378] FINISH, SetMOMPolicyParametersVDSCommand, log id: 4161db52 >2014-08-07 05:32:18,344 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-94) [6e904378] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host rack was set to Up. >2014-08-07 05:32:18,352 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-94) [6e904378] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host rack. >2014-08-07 05:32:18,381 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-94) [a61d6df] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS >2014-08-07 05:32:24,503 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-61) hostFromVds::selectedVds - rack, spmStatus Free, storage pool Default >2014-08-07 05:32:24,508 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-61) starting spm on vds rack, storage pool Default, prevId -1, LVER -1 >2014-08-07 05:32:24,510 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-61) START, SpmStartVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, prevId=-1, prevLVER=-1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: d84dfa5 >2014-08-07 05:32:24,559 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-61) spmStart polling started: taskId = 95bb7504-b263-4a49-8fd5-5810b3073302 >2014-08-07 05:32:25,566 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-61) spmStart polling ended: taskId = 95bb7504-b263-4a49-8fd5-5810b3073302 task status = finished >2014-08-07 05:32:25,621 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-61) spmStart polling ended, spm status: SPM >2014-08-07 05:32:25,628 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-61) START, HSMClearTaskVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, taskId=95bb7504-b263-4a49-8fd5-5810b3073302), log id: 315052a >2014-08-07 05:32:25,674 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-61) FINISH, HSMClearTaskVDSCommand, log id: 315052a >2014-08-07 05:32:25,674 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-61) FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@29ccb33f, log id: d84dfa5 >2014-08-07 05:32:25,679 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-61) Initialize Irs proxy from vds: 10.34.62.208 >2014-08-07 05:32:25,686 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-61) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208). >2014-08-07 05:32:25,699 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-44) START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 2ac73b48 >2014-08-07 05:32:25,732 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-44) -- executeIrsBrokerCommand: Attempting on storage pool 00000002-0002-0002-0002-0000000001d9 >2014-08-07 05:32:25,735 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-44) START, HSMGetAllTasksInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 7e12fcdb >2014-08-07 05:32:25,742 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-44) FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 7e12fcdb >2014-08-07 05:32:25,744 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-44) FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 2ac73b48 >2014-08-07 05:32:25,745 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (org.ovirt.thread.pool-8-thread-44) Discovered no tasks on Storage Pool Default >2014-08-07 06:30:54,958 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-23) Principal admin::internal synchronized >2014-08-07 06:30:55,693 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-93) Attempting to update VMs/Templates Ovf. >2014-08-07 07:30:54,968 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-30) Principal admin::internal synchronized >2014-08-07 07:30:55,709 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-52) Attempting to update VMs/Templates Ovf. >2014-08-07 08:30:54,976 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-25) Principal admin::internal synchronized >2014-08-07 08:30:55,743 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-40) Attempting to update VMs/Templates Ovf. >2014-08-07 08:32:01,433 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-41) domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 in problem. vds: rack >2014-08-07 08:32:16,555 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-4) domain 44559ec0-6bef-491c-bbd6-e67278f37987:export in problem. vds: rack >2014-08-07 08:33:39,052 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-70) Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2014-08-07 08:33:39,059 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-70) [57c471e1] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-08-07 08:33:39,062 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-70) [57c471e1] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Active to Unknown, reason : null >2014-08-07 08:33:39,065 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-70) [57c471e1] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Active to Unknown, reason : null >2014-08-07 08:33:39,083 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-70) [57c471e1] Correlation ID: 57c471e1, Call Stack: null, Custom Event ID: -1, Message: Invalid status on Data Center Default. Setting Data Center status to Non Responsive (On host rack, Error: Cannot access master storage domain). >2014-08-07 08:35:39,103 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-70) [57c471e1] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2014-08-07 08:35:39,105 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-70) [57c471e1] hostFromVds::selectedVds - rack, spmStatus returned null! >2014-08-07 08:35:39,106 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-70) [57c471e1] IrsBroker::Failed::GetStoragePoolInfoVDS >2014-08-07 08:35:39,108 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-70) [57c471e1] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2014-08-07 08:35:39,153 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-9) [5e9ac41c] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage >2014-08-07 08:35:39,176 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-9) [5e9ac41c] START, ResetIrsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, ignoreStopFailed = true), log id: d035289 >2014-08-07 08:35:39,180 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-9) [5e9ac41c] START, SpmStopVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9), log id: 6c90db65 >2014-08-07 08:35:39,225 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-9) [5e9ac41c] SpmStopVDSCommand::Stopping SPM on vds rack, pool id 00000002-0002-0002-0002-0000000001d9 >2014-08-07 08:37:39,075 WARN [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-9) [5e9ac41c] could not stop spm of pool 00000002-0002-0002-0002-0000000001d9 on vds 17f7fe56-5f32-40ef-b098-611b84484e0b - reason: org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcRunTimeException: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException >2014-08-07 08:37:39,080 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-9) [5e9ac41c] FINISH, SpmStopVDSCommand, log id: 6c90db65 >2014-08-07 08:37:39,084 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-61) Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: VDSNetworkException: org.apache.commons.httpclient.NoHttpResponseException: The server 10.34.62.208 failed to respond >2014-08-07 08:37:39,089 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-9) [5e9ac41c] FINISH, ResetIrsVDSCommand, log id: d035289 >2014-08-07 08:37:39,087 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ListVDSCommand] (DefaultQuartzScheduler_Worker-3) Command ListVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-08-07 08:37:39,099 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-3) Host rack is not responding. It will stay in Connecting state for a grace period of $80 seconds and after that an attempt to fence the host will be issued. >2014-08-07 08:37:39,103 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-61) hostFromVds::selectedVds - rack, spmStatus returned null! >2014-08-07 08:37:39,101 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-3) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:407) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.fetchRunningVms(VdsUpdateRunTimeInfo.java:971) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVmStats(VdsUpdateRunTimeInfo.java:920) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:638) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor17.invoke(Unknown Source) [:1.7.0_55] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_55] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_55] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_55] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_55] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_55] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_55] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_55] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.list(Unknown Source) > at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source) [:1.7.0_55] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_55] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_55] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_55] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_55] > >2014-08-07 08:37:39,121 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-9) [5e9ac41c] Correlation ID: 5e9ac41c, Job ID: a99e3afe-91a6-425b-bdd4-d7c034ab6dc7, Call Stack: null, Custom Event ID: -1, Message: Failed to Reconstruct Master Domain for Data Center Default. >2014-08-07 08:37:39,152 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-29) [57c471e1] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue >2014-08-07 08:37:39,154 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-40) starting processDomainRecovery for domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 >2014-08-07 08:37:39,163 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-3) Failed to refresh VDS , vds = 17f7fe56-5f32-40ef-b098-611b84484e0b : rack, VDS Network Error, continuing. >java.net.ConnectException: Connection refused >2014-08-07 08:37:39,227 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-40) Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 was reported by all hosts in status UP as problematic. Not moving the domain to NonOperational because it is being reconstructed now. >2014-08-07 08:37:39,247 WARN [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-40) [f2a2b4e] CanDoAction of action ReconstructMasterDomain failed. Reasons:VAR__ACTION__RECONSTRUCT_MASTER,VAR__TYPE__STORAGE__DOMAIN,ACTION_TYPE_FAILED_NO_VDS_IN_POOL >2014-08-07 08:37:39,254 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-40) [f2a2b4e] Removing vds [] from the domain in maintenance cache >2014-08-07 08:37:39,255 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-40) [f2a2b4e] Removing host(s) [] from hosts unseen domain report cache >2014-08-07 08:37:39,255 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-29) [57c471e1] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue >2014-08-07 08:37:39,257 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-33) starting processDomainRecovery for domain 44559ec0-6bef-491c-bbd6-e67278f37987:export >2014-08-07 08:37:39,262 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-33) Domain 44559ec0-6bef-491c-bbd6-e67278f37987:export was reported by all hosts in status UP as problematic. Moving the domain to NonOperational. >2014-08-07 08:37:39,279 INFO [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-33) [3915e0d8] Lock Acquired to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE >, sharedLocks= ] >2014-08-07 08:37:39,290 INFO [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-33) [3915e0d8] Running command: DeactivateStorageDomainCommand internal: true. Entities affected : ID: 44559ec0-6bef-491c-bbd6-e67278f37987 Type: Storage >2014-08-07 08:37:39,302 INFO [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-33) [3915e0d8] Lock freed to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE >, sharedLocks= ] >2014-08-07 08:37:39,316 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-33) [3915e0d8] Correlation ID: 3915e0d8, Job ID: a49c7a54-2313-4558-9f1b-09202c86e30a, Call Stack: null, Custom Event ID: -1, Message: Storage Domain export (Data Center Default) was deactivated by system because it's not visible by any of the hosts. >2014-08-07 08:37:39,321 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-33) [3915e0d8] Removing vds [] from the domain in maintenance cache >2014-08-07 08:37:39,322 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-33) [3915e0d8] Removing host(s) [] from hosts unseen domain report cache >2014-08-07 08:37:42,254 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-89) Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@40b42e09 >2014-08-07 08:37:42,255 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-89) HostName = rack >2014-08-07 08:37:42,256 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-89) Command GetCapabilitiesVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]) execution failed. Exception: VDSRecoveringException: Recovering from crash or Initializing >2014-08-07 08:37:42,275 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-89) Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSRecoveringException: Recovering from crash or Initializing > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:16) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor17.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Host rack is initializing. Message: Recovering from crash or Initializing >2014-08-07 08:37:42,282 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-89) Failed to refresh VDS , vds = 17f7fe56-5f32-40ef-b098-611b84484e0b : rack, error = Recovering from crash or Initializing, continuing. >2014-08-07 08:37:45,521 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-86) START, GetHardwareInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]), log id: 70520d0a >2014-08-07 08:37:45,572 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-86) FINISH, GetHardwareInfoVDSCommand, log id: 70520d0a >2014-08-07 08:37:45,614 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-86) [384faa36] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS >2014-08-07 08:37:45,624 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-86) [384faa36] Cancelling the recovery from crash timer for VDS 17f7fe56-5f32-40ef-b098-611b84484e0b because vds started initializing >2014-08-07 08:37:45,626 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-86) [20c96164] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-08-07 08:37:45,641 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-86) [7b8ce667] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-08-07 08:37:45,654 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-86) [7b8ce667] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, storageType = NFS, connectionList = [{ id: 05047c9a-152e-4ef2-b118-4a1753c1eae0, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35export, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 55683a48 >2014-08-07 08:37:45,718 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-86) [7b8ce667] FINISH, ConnectStorageServerVDSCommand, return: {5083f370-af09-457b-9b73-3440e8b65a07=0, 05047c9a-152e-4ef2-b118-4a1753c1eae0=0}, log id: 55683a48 >2014-08-07 08:37:45,719 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-86) [7b8ce667] Host rack storage connection was succeeded >2014-08-07 08:37:45,728 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-18) START, ConnectStoragePoolVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, masterVersion = 1), log id: 45531e49 >2014-08-07 08:37:49,113 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-17) Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Inactive to Unknown, reason : no reporting hosts >2014-08-07 08:37:49,118 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-17) Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Inactive to Unknown, reason : no reporting hosts >2014-08-07 08:38:46,161 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-18) FINISH, ConnectStoragePoolVDSCommand, log id: 45531e49 >2014-08-07 08:38:46,175 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-86) [7b8ce667] START, SetMOMPolicyParametersVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 4f2ecd20 >2014-08-07 08:38:46,269 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-86) [7b8ce667] FINISH, SetMOMPolicyParametersVDSCommand, log id: 4f2ecd20 >2014-08-07 08:38:46,275 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-86) [7b8ce667] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host rack was set to Up. >2014-08-07 08:38:46,283 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-86) [7b8ce667] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host rack. >2014-08-07 08:38:46,303 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-86) [5ff62557] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS >2014-08-07 08:38:49,238 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-14) [2bb2f45d] hostFromVds::selectedVds - rack, spmStatus Free, storage pool Default >2014-08-07 08:38:49,244 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-14) [2bb2f45d] starting spm on vds rack, storage pool Default, prevId -1, LVER -1 >2014-08-07 08:38:49,246 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-14) [2bb2f45d] START, SpmStartVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, prevId=-1, prevLVER=-1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 4424f251 >2014-08-07 08:38:49,294 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-14) [2bb2f45d] spmStart polling started: taskId = f089e1c2-6576-4d86-9f38-6fedbb8dc7f3 >2014-08-07 08:38:50,301 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-14) [2bb2f45d] spmStart polling ended: taskId = f089e1c2-6576-4d86-9f38-6fedbb8dc7f3 task status = finished >2014-08-07 08:38:50,356 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-14) [2bb2f45d] spmStart polling ended, spm status: SPM >2014-08-07 08:38:50,359 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-14) [2bb2f45d] START, HSMClearTaskVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, taskId=f089e1c2-6576-4d86-9f38-6fedbb8dc7f3), log id: 2946eb35 >2014-08-07 08:38:50,405 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-14) [2bb2f45d] FINISH, HSMClearTaskVDSCommand, log id: 2946eb35 >2014-08-07 08:38:50,406 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-14) [2bb2f45d] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@7a10977a, log id: 4424f251 >2014-08-07 08:38:50,410 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-14) [2bb2f45d] Initialize Irs proxy from vds: 10.34.62.208 >2014-08-07 08:38:50,420 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-14) [2bb2f45d] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208). >2014-08-07 08:38:50,432 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-3) [2bb2f45d] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 1e9994ae >2014-08-07 08:38:50,463 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-3) [2bb2f45d] -- executeIrsBrokerCommand: Attempting on storage pool 00000002-0002-0002-0002-0000000001d9 >2014-08-07 08:38:50,467 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-3) [2bb2f45d] START, HSMGetAllTasksInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 405a044b >2014-08-07 08:38:50,473 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-3) [2bb2f45d] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 405a044b >2014-08-07 08:38:50,474 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-3) [2bb2f45d] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 1e9994ae >2014-08-07 08:38:50,476 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (org.ovirt.thread.pool-8-thread-3) [2bb2f45d] Discovered no tasks on Storage Pool Default >2014-08-07 09:29:22,699 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-2) Running command: LoginUserCommand internal: false. >2014-08-07 09:29:22,705 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-2) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-08-07 09:30:54,982 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-20) Principal admin::internal synchronized >2014-08-07 09:30:55,758 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-25) Attempting to update VMs/Templates Ovf. >2014-08-07 09:53:13,531 INFO [org.ovirt.engine.core.bll.aaa.LoginAdminUserCommand] (ajp--127.0.0.1-8702-6) Running command: LoginAdminUserCommand internal: false. >2014-08-07 09:53:13,541 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-08-07 09:55:18,568 INFO [org.ovirt.engine.core.bll.RunVmCommand] (ajp--127.0.0.1-8702-3) [59a10b9a] Lock Acquired to object EngineLock [exclusiveLocks= key: 9042bb21-3f27-48bb-ab26-c0d8b18c840d value: VM >, sharedLocks= ] >2014-08-07 09:55:18,779 INFO [org.ovirt.engine.core.bll.RunVmCommand] (org.ovirt.thread.pool-8-thread-12) [59a10b9a] Running command: RunVmCommand internal: false. Entities affected : ID: 9042bb21-3f27-48bb-ab26-c0d8b18c840d Type: VM >2014-08-07 09:55:18,786 INFO [org.ovirt.engine.core.vdsbroker.ResumeVDSCommand] (org.ovirt.thread.pool-8-thread-12) [59a10b9a] START, ResumeVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vmId=9042bb21-3f27-48bb-ab26-c0d8b18c840d), log id: 626248c8 >2014-08-07 09:55:18,790 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ResumeBrokerVDSCommand] (org.ovirt.thread.pool-8-thread-12) [59a10b9a] START, ResumeBrokerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vmId=9042bb21-3f27-48bb-ab26-c0d8b18c840d), log id: 1c5c5f8a >2014-08-07 09:55:18,844 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ResumeBrokerVDSCommand] (org.ovirt.thread.pool-8-thread-12) [59a10b9a] FINISH, ResumeBrokerVDSCommand, log id: 1c5c5f8a >2014-08-07 09:55:18,845 INFO [org.ovirt.engine.core.vdsbroker.ResumeVDSCommand] (org.ovirt.thread.pool-8-thread-12) [59a10b9a] FINISH, ResumeVDSCommand, return: PoweringUp, log id: 626248c8 >2014-08-07 09:55:18,846 INFO [org.ovirt.engine.core.bll.RunVmCommand] (org.ovirt.thread.pool-8-thread-12) [59a10b9a] Lock freed to object EngineLock [exclusiveLocks= key: 9042bb21-3f27-48bb-ab26-c0d8b18c840d value: VM >, sharedLocks= ] >2014-08-07 09:55:18,908 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-12) [59a10b9a] Correlation ID: 59a10b9a, Job ID: e5f550ac-cbe6-4c36-b9dd-0d8f7f5a84f5, Call Stack: null, Custom Event ID: -1, Message: VM ovirtnode was resumed by admin (Host: rack). >2014-08-07 09:55:21,273 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-45) VM ovirtnode 9042bb21-3f27-48bb-ab26-c0d8b18c840d moved from Paused --> Up >2014-08-07 09:55:21,302 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-45) Correlation ID: 59a10b9a, Job ID: e5f550ac-cbe6-4c36-b9dd-0d8f7f5a84f5, Call Stack: null, Custom Event ID: -1, Message: VM ovirtnode started on Host rack >2014-08-07 09:55:30,768 INFO [org.ovirt.engine.core.bll.RunVmCommand] (ajp--127.0.0.1-8702-6) [4c3d5594] Lock Acquired to object EngineLock [exclusiveLocks= key: 78722009-2727-4715-91bf-dbbb33ce111e value: VM >, sharedLocks= ] >2014-08-07 09:55:30,960 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (ajp--127.0.0.1-8702-6) [4c3d5594] START, IsVmDuringInitiatingVDSCommand( vmId = 78722009-2727-4715-91bf-dbbb33ce111e), log id: d412eca >2014-08-07 09:55:30,962 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (ajp--127.0.0.1-8702-6) [4c3d5594] FINISH, IsVmDuringInitiatingVDSCommand, return: false, log id: d412eca >2014-08-07 09:55:31,063 INFO [org.ovirt.engine.core.bll.RunVmCommand] (org.ovirt.thread.pool-8-thread-11) [4c3d5594] Running command: RunVmCommand internal: false. Entities affected : ID: 78722009-2727-4715-91bf-dbbb33ce111e Type: VM >2014-08-07 09:55:31,119 INFO [org.ovirt.engine.core.vdsbroker.UpdateVmDynamicDataVDSCommand] (org.ovirt.thread.pool-8-thread-11) [4c3d5594] START, UpdateVmDynamicDataVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vmDynamic=org.ovirt.engine.core.common.businessentities.VmDynamic@cd426e7), log id: 7f00aeb >2014-08-07 09:55:31,133 INFO [org.ovirt.engine.core.vdsbroker.UpdateVmDynamicDataVDSCommand] (org.ovirt.thread.pool-8-thread-11) [4c3d5594] FINISH, UpdateVmDynamicDataVDSCommand, log id: 7f00aeb >2014-08-07 09:55:31,170 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (org.ovirt.thread.pool-8-thread-11) [4c3d5594] START, CreateVmVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vmId=78722009-2727-4715-91bf-dbbb33ce111e, vm=VM [testtesttest]), log id: d879f9e >2014-08-07 09:55:31,204 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (org.ovirt.thread.pool-8-thread-11) [4c3d5594] START, CreateVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vmId=78722009-2727-4715-91bf-dbbb33ce111e, vm=VM [testtesttest]), log id: 6b44275 >2014-08-07 09:55:31,311 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (org.ovirt.thread.pool-8-thread-11) [4c3d5594] org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand kvmEnable=true,keyboardLayout=en-us,nice=0,displayNetwork=ovirtmgmt,copyPasteEnable=true,timeOffset=0,transparentHugePages=true,vmId=78722009-2727-4715-91bf-dbbb33ce111e,acpiEnable=true,custom={device_069719e3-08a8-492f-9e06-1d26df82901adevice_6d1bd3c9-c40c-41fe-b152-d7a6be5f40ffdevice_ae661712-04dc-40c7-bb5d-ac71c44882b6=VmDevice {vmId=78722009-2727-4715-91bf-dbbb33ce111e, deviceId=ae661712-04dc-40c7-bb5d-ac71c44882b6, device=unix, type=CHANNEL, bootOrder=0, specParams={}, address={port=1, bus=0, controller=0, type=virtio-serial}, managed=false, plugged=true, readOnly=false, deviceAlias=channel0, customProperties={}, snapshotId=null}, device_069719e3-08a8-492f-9e06-1d26df82901adevice_6d1bd3c9-c40c-41fe-b152-d7a6be5f40ffdevice_ae661712-04dc-40c7-bb5d-ac71c44882b6device_710595e4-0cbc-4ae3-ae8d-3c123dad65cb=VmDevice {vmId=78722009-2727-4715-91bf-dbbb33ce111e, deviceId=710595e4-0cbc-4ae3-ae8d-3c123dad65cb, device=unix, type=CHANNEL, bootOrder=0, specParams={}, address={port=2, bus=0, controller=0, type=virtio-serial}, managed=false, plugged=true, readOnly=false, deviceAlias=channel1, customProperties={}, snapshotId=null}, device_069719e3-08a8-492f-9e06-1d26df82901adevice_6d1bd3c9-c40c-41fe-b152-d7a6be5f40ff=VmDevice {vmId=78722009-2727-4715-91bf-dbbb33ce111e, deviceId=6d1bd3c9-c40c-41fe-b152-d7a6be5f40ff, device=virtio-serial, type=CONTROLLER, bootOrder=0, specParams={}, address={bus=0x00, domain=0x0000, type=pci, slot=0x05, function=0x0}, managed=false, plugged=true, readOnly=false, deviceAlias=virtio-serial0, customProperties={}, snapshotId=null}, device_069719e3-08a8-492f-9e06-1d26df82901a=VmDevice {vmId=78722009-2727-4715-91bf-dbbb33ce111e, deviceId=069719e3-08a8-492f-9e06-1d26df82901a, device=ide, type=CONTROLLER, bootOrder=0, specParams={}, address={bus=0x00, domain=0x0000, type=pci, slot=0x01, function=0x1}, managed=false, plugged=true, readOnly=false, deviceAlias=ide0, customProperties={}, snapshotId=null}, device_069719e3-08a8-492f-9e06-1d26df82901adevice_6d1bd3c9-c40c-41fe-b152-d7a6be5f40ffdevice_ae661712-04dc-40c7-bb5d-ac71c44882b6device_710595e4-0cbc-4ae3-ae8d-3c123dad65cbdevice_1ea1dd78-6851-4585-8184-ea9b5bd84156=VmDevice {vmId=78722009-2727-4715-91bf-dbbb33ce111e, deviceId=1ea1dd78-6851-4585-8184-ea9b5bd84156, device=spicevmc, type=CHANNEL, bootOrder=0, specParams={}, address={port=3, bus=0, controller=0, type=virtio-serial}, managed=false, plugged=true, readOnly=false, deviceAlias=channel2, customProperties={}, snapshotId=null}},spiceSslCipherSuite=DEFAULT,memSize=5120,smp=1,emulatedMachine=rhel6.5.0,vmType=kvm,memGuaranteedSize=5120,display=qxl,smartcardEnable=false,bootMenuEnable=false,spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,susbredir,ssmartcard,smpCoresPerSocket=1,numaTune={mode=preferred},maxVCpus=160,devices=[{address={bus=0x00, domain=0x0000, slot=0x02, type=pci, function=0x0}, specParams={vram=32768, heads=1}, device=qxl, type=video, deviceId=d962101d-abb7-4fbb-b249-2abdf8b3ee46}, {shared=false, iface=ide, index=2, address={unit=0, bus=1, target=0, controller=0, type=drive}, specParams={path=}, path=, device=cdrom, type=disk, readonly=true, deviceId=a0db1b5f-d569-4b3f-a0e3-3c2992216d80}, {shared=false, index=0, volumeID=b05d9fa4-2cf0-414e-b031-f89fcdf2cf0b, propagateErrors=off, format=raw, type=disk, iface=virtio, address={bus=0x00, domain=0x0000, slot=0x06, type=pci, function=0x0}, domainID=71f82fcd-4fa6-4f56-b228-f4fa4bd4f752, imageID=6cae69b6-2d4f-4e61-993c-dce113dd19e5, specParams={}, optional=false, device=disk, poolID=00000002-0002-0002-0002-0000000001d9, readonly=false, deviceId=6cae69b6-2d4f-4e61-993c-dce113dd19e5}, {nicModel=pv, address={bus=0x00, domain=0x0000, slot=0x03, type=pci, function=0x0}, specParams={outbound={}, inbound={}}, macAddr=00:1a:4a:ae:a8:00, device=bridge, linkActive=true, type=interface, filter=vdsm-no-mac-spoofing, network=ovirtmgmt, deviceId=d00434c2-b72b-4572-9f28-162bfe749013}, {index=0, model=virtio-scsi, address={bus=0x00, domain=0x0000, slot=0x04, type=pci, function=0x0}, specParams={}, device=scsi, type=controller, deviceId=66902ac6-2baf-478d-821c-71f53156d266}],vmName=testtesttest,cpuType=Westmere,fileTransferEnable=true >2014-08-07 09:55:31,362 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (org.ovirt.thread.pool-8-thread-11) [4c3d5594] FINISH, CreateVDSCommand, log id: 6b44275 >2014-08-07 09:55:31,374 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (org.ovirt.thread.pool-8-thread-11) [4c3d5594] FINISH, CreateVmVDSCommand, return: WaitForLaunch, log id: d879f9e >2014-08-07 09:55:31,374 INFO [org.ovirt.engine.core.bll.RunVmCommand] (org.ovirt.thread.pool-8-thread-11) [4c3d5594] Lock freed to object EngineLock [exclusiveLocks= key: 78722009-2727-4715-91bf-dbbb33ce111e value: VM >, sharedLocks= ] >2014-08-07 09:55:31,387 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-11) [4c3d5594] Correlation ID: 4c3d5594, Job ID: d0b854c7-90ae-4139-b7eb-cc2985d28c61, Call Stack: null, Custom Event ID: -1, Message: VM testtesttest was started by admin (Host: rack). >2014-08-07 09:55:33,421 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-36) VM testtesttest 78722009-2727-4715-91bf-dbbb33ce111e moved from WaitForLaunch --> PoweringUp >2014-08-07 09:55:33,426 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVdsCommand] (DefaultQuartzScheduler_Worker-36) START, FullListVdsCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b], vmIds=[78722009-2727-4715-91bf-dbbb33ce111e]), log id: f34781f >2014-08-07 09:55:33,434 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVdsCommand] (DefaultQuartzScheduler_Worker-36) FINISH, FullListVdsCommand, return: [{displaySecurePort=5903, kvmEnable=true, nicModel=rtl8139,pv, keyboardLayout=en-us, displayIp=10.34.62.208, pauseCode=NOERR, nice=0, displayNetwork=ovirtmgmt, copyPasteEnable=true, timeOffset=0, transparentHugePages=true, vmId=78722009-2727-4715-91bf-dbbb33ce111e, acpiEnable=true, custom={device_069719e3-08a8-492f-9e06-1d26df82901adevice_6d1bd3c9-c40c-41fe-b152-d7a6be5f40ffdevice_ae661712-04dc-40c7-bb5d-ac71c44882b6=VmDevice {vmId=78722009-2727-4715-91bf-dbbb33ce111e, deviceId=ae661712-04dc-40c7-bb5d-ac71c44882b6, device=unix, type=CHANNEL, bootOrder=0, specParams={}, address={port=1, bus=0, controller=0, type=virtio-serial}, managed=false, plugged=true, readOnly=false, deviceAlias=channel0, customProperties={}, snapshotId=null}, device_069719e3-08a8-492f-9e06-1d26df82901adevice_6d1bd3c9-c40c-41fe-b152-d7a6be5f40ffdevice_ae661712-04dc-40c7-bb5d-ac71c44882b6device_710595e4-0cbc-4ae3-ae8d-3c123dad65cb=VmDevice {vmId=78722009-2727-4715-91bf-dbbb33ce111e, deviceId=710595e4-0cbc-4ae3-ae8d-3c123dad65cb, device=unix, type=CHANNEL, bootOrder=0, specParams={}, address={port=2, bus=0, controller=0, type=virtio-serial}, managed=false, plugged=true, readOnly=false, deviceAlias=channel1, customProperties={}, snapshotId=null}, device_069719e3-08a8-492f-9e06-1d26df82901a=VmDevice {vmId=78722009-2727-4715-91bf-dbbb33ce111e, deviceId=069719e3-08a8-492f-9e06-1d26df82901a, device=ide, type=CONTROLLER, bootOrder=0, specParams={}, address={bus=0x00, domain=0x0000, type=pci, slot=0x01, function=0x1}, managed=false, plugged=true, readOnly=false, deviceAlias=ide0, customProperties={}, snapshotId=null}, device_069719e3-08a8-492f-9e06-1d26df82901adevice_6d1bd3c9-c40c-41fe-b152-d7a6be5f40ff=VmDevice {vmId=78722009-2727-4715-91bf-dbbb33ce111e, deviceId=6d1bd3c9-c40c-41fe-b152-d7a6be5f40ff, device=virtio-serial, type=CONTROLLER, bootOrder=0, specParams={}, address={bus=0x00, domain=0x0000, type=pci, slot=0x05, function=0x0}, managed=false, plugged=true, readOnly=false, deviceAlias=virtio-serial0, customProperties={}, snapshotId=null}, device_069719e3-08a8-492f-9e06-1d26df82901adevice_6d1bd3c9-c40c-41fe-b152-d7a6be5f40ffdevice_ae661712-04dc-40c7-bb5d-ac71c44882b6device_710595e4-0cbc-4ae3-ae8d-3c123dad65cbdevice_1ea1dd78-6851-4585-8184-ea9b5bd84156=VmDevice {vmId=78722009-2727-4715-91bf-dbbb33ce111e, deviceId=1ea1dd78-6851-4585-8184-ea9b5bd84156, device=spicevmc, type=CHANNEL, bootOrder=0, specParams={}, address={port=3, bus=0, controller=0, type=virtio-serial}, managed=false, plugged=true, readOnly=false, deviceAlias=channel2, customProperties={}, snapshotId=null}}, spiceSslCipherSuite=DEFAULT, memSize=5120, smp=1, displayPort=5902, emulatedMachine=rhel6.5.0, vmType=kvm, status=Up, memGuaranteedSize=5120, display=qxl, pid=8661, smartcardEnable=false, bootMenuEnable=false, spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,susbredir,ssmartcard, numaTune={mode=preferred}, smpCoresPerSocket=1, maxVCpus=160, clientIp=, devices=[Ljava.lang.Object;@3dba4305, vmName=testtesttest, fileTransferEnable=true, cpuType=Westmere}], log id: f34781f >2014-08-07 09:55:33,453 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-36) Received a memballoon Device without an address when processing VM 78722009-2727-4715-91bf-dbbb33ce111e devices, skipping device: {alias=balloon0, specParams={model=none}, device=memballoon, type=balloon} >2014-08-07 09:55:33,454 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-36) Received a spice Device without an address when processing VM 78722009-2727-4715-91bf-dbbb33ce111e devices, skipping device: {port=5902, specParams={spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,susbredir,ssmartcard, keyMap=en-us, displayNetwork=ovirtmgmt, copyPasteEnable=true, displayIp=10.34.62.208}, device=spice, tlsPort=5903, type=graphics} >2014-08-07 09:55:38,935 INFO [org.ovirt.engine.core.bll.StopVmCommand] (org.ovirt.thread.pool-8-thread-13) [7b2c4cab] Running command: StopVmCommand internal: false. Entities affected : ID: 78722009-2727-4715-91bf-dbbb33ce111e Type: VM >2014-08-07 09:55:38,943 INFO [org.ovirt.engine.core.vdsbroker.DestroyVmVDSCommand] (org.ovirt.thread.pool-8-thread-13) [7b2c4cab] START, DestroyVmVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vmId=78722009-2727-4715-91bf-dbbb33ce111e, force=false, secondsToWait=0, gracefully=false, reason=), log id: 5984d7be >2014-08-07 09:55:38,953 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (org.ovirt.thread.pool-8-thread-13) [7b2c4cab] START, DestroyVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vmId=78722009-2727-4715-91bf-dbbb33ce111e, force=false, secondsToWait=0, gracefully=false, reason=), log id: 2ca735c0 >2014-08-07 09:55:39,449 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (org.ovirt.thread.pool-8-thread-13) [7b2c4cab] FINISH, DestroyVDSCommand, log id: 2ca735c0 >2014-08-07 09:55:39,457 INFO [org.ovirt.engine.core.vdsbroker.DestroyVmVDSCommand] (org.ovirt.thread.pool-8-thread-13) [7b2c4cab] FINISH, DestroyVmVDSCommand, return: Down, log id: 5984d7be >2014-08-07 09:55:39,465 INFO [org.ovirt.engine.core.bll.ProcessDownVmCommand] (org.ovirt.thread.pool-8-thread-16) [797ae1f9] Running command: ProcessDownVmCommand internal: true. >2014-08-07 09:55:39,483 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-13) [7b2c4cab] Correlation ID: 7b2c4cab, Job ID: 040e177f-3ede-4d2f-b2f9-6ce8e4c173b5, Call Stack: null, Custom Event ID: -1, Message: VM testtesttest powered off by admin (Host: rack) (Reason: Not Specified). >2014-08-07 10:17:39,711 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-7) Running command: LoginUserCommand internal: false. >2014-08-07 10:17:39,716 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-7) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-08-07 10:30:54,990 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-95) Principal admin::internal synchronized >2014-08-07 10:30:55,773 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-87) Attempting to update VMs/Templates Ovf. >2014-08-07 10:52:55,123 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-6) Running command: LoginUserCommand internal: false. >2014-08-07 10:52:55,131 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-08-07 10:56:37,775 INFO [org.ovirt.engine.core.bll.storage.AddEmptyStoragePoolCommand] (ajp--127.0.0.1-8702-9) [579bab3a] Running command: AddEmptyStoragePoolCommand internal: false. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2014-08-07 10:56:37,808 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-9) [579bab3a] Correlation ID: 579bab3a, Call Stack: null, Custom Event ID: -1, Message: Data Center local, Compatibility Version 3.5 and Quota Type DISABLED was added by admin >2014-08-07 10:56:52,943 INFO [org.ovirt.engine.core.bll.AddVdsGroupCommand] (ajp--127.0.0.1-8702-9) [3cf6d2dc] Running command: AddVdsGroupCommand internal: false. Entities affected : ID: 29cf5221-8594-4e8b-afd3-bb0227304f01 Type: StoragePool >2014-08-07 10:56:52,990 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-9) [3cf6d2dc] Correlation ID: 3cf6d2dc, Call Stack: null, Custom Event ID: -1, Message: Cluster local was added by admin >2014-08-07 10:57:34,620 INFO [org.ovirt.engine.core.bll.MaintenanceNumberOfVdssCommand] (ajp--127.0.0.1-8702-1) [2438c510] Lock Acquired to object EngineLock [exclusiveLocks= , sharedLocks= key: 00000002-0002-0002-0002-0000000001d9 value: POOL >] >2014-08-07 10:57:34,694 WARN [org.ovirt.engine.core.bll.MaintenanceNumberOfVdssCommand] (ajp--127.0.0.1-8702-1) [2438c510] CanDoAction of action MaintenanceNumberOfVdss failed. Reasons:VAR__TYPE__HOST,VAR__ACTION__MAINTENANCE,CANNOT_MAINTENANCE_VDS_RUN_VMS_NO_OTHER_RUNNING_VDS,$ClustersList Default,$HostsList rack >2014-08-07 10:57:34,696 INFO [org.ovirt.engine.core.bll.MaintenanceNumberOfVdssCommand] (ajp--127.0.0.1-8702-1) [2438c510] Lock freed to object EngineLock [exclusiveLocks= , sharedLocks= key: 00000002-0002-0002-0002-0000000001d9 value: POOL >] >2014-08-07 10:57:51,443 INFO [org.ovirt.engine.core.bll.StopVmCommand] (org.ovirt.thread.pool-8-thread-47) [20541247] Running command: StopVmCommand internal: false. Entities affected : ID: 9042bb21-3f27-48bb-ab26-c0d8b18c840d Type: VM >2014-08-07 10:57:51,448 INFO [org.ovirt.engine.core.vdsbroker.DestroyVmVDSCommand] (org.ovirt.thread.pool-8-thread-47) [20541247] START, DestroyVmVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vmId=9042bb21-3f27-48bb-ab26-c0d8b18c840d, force=false, secondsToWait=0, gracefully=false, reason=), log id: 402666e2 >2014-08-07 10:57:51,455 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (org.ovirt.thread.pool-8-thread-47) [20541247] START, DestroyVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vmId=9042bb21-3f27-48bb-ab26-c0d8b18c840d, force=false, secondsToWait=0, gracefully=false, reason=), log id: 15e94ca6 >2014-08-07 10:57:51,954 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (org.ovirt.thread.pool-8-thread-47) [20541247] FINISH, DestroyVDSCommand, log id: 15e94ca6 >2014-08-07 10:57:51,962 INFO [org.ovirt.engine.core.vdsbroker.DestroyVmVDSCommand] (org.ovirt.thread.pool-8-thread-47) [20541247] FINISH, DestroyVmVDSCommand, return: Down, log id: 402666e2 >2014-08-07 10:57:51,969 INFO [org.ovirt.engine.core.bll.ProcessDownVmCommand] (org.ovirt.thread.pool-8-thread-39) [2b8e1344] Running command: ProcessDownVmCommand internal: true. >2014-08-07 10:57:51,987 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-47) [20541247] Correlation ID: 20541247, Job ID: fb472bec-4098-4884-a42f-3071da83f9e8, Call Stack: null, Custom Event ID: -1, Message: VM ovirtnode powered off by admin (Host: rack) (Reason: Not Specified). >2014-08-07 10:58:17,194 INFO [org.ovirt.engine.core.bll.MaintenanceNumberOfVdssCommand] (ajp--127.0.0.1-8702-5) [4306be4f] Lock Acquired to object EngineLock [exclusiveLocks= , sharedLocks= key: 00000002-0002-0002-0002-0000000001d9 value: POOL >] >2014-08-07 10:58:17,227 INFO [org.ovirt.engine.core.bll.MaintenanceNumberOfVdssCommand] (org.ovirt.thread.pool-8-thread-10) [4306be4f] Running command: MaintenanceNumberOfVdssCommand internal: false. Entities affected : ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS >2014-08-07 10:58:17,232 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-8-thread-10) [4306be4f] START, SetVdsStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, status=PreparingForMaintenance, nonOperationalReason=NONE, stopSpmFailureLogged=true), log id: 801e5d2 >2014-08-07 10:58:17,236 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-8-thread-10) [4306be4f] VDS rack is spm and moved from up calling resetIrs. >2014-08-07 10:58:17,239 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-10) [4306be4f] START, ResetIrsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, ignoreStopFailed = false), log id: 7c3bbc08 >2014-08-07 10:58:17,244 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-10) [4306be4f] START, SpmStopVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9), log id: 294002ca >2014-08-07 10:58:17,252 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-10) [4306be4f] SpmStopVDSCommand::Stopping SPM on vds rack, pool id 00000002-0002-0002-0002-0000000001d9 >2014-08-07 10:58:17,316 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-10) [4306be4f] FINISH, SpmStopVDSCommand, log id: 294002ca >2014-08-07 10:58:17,322 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-10) [4306be4f] FINISH, ResetIrsVDSCommand, log id: 7c3bbc08 >2014-08-07 10:58:17,329 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-8-thread-10) [4306be4f] FINISH, SetVdsStatusVDSCommand, log id: 801e5d2 >2014-08-07 10:58:17,390 INFO [org.ovirt.engine.core.bll.MaintenanceNumberOfVdssCommand] (org.ovirt.thread.pool-8-thread-10) [4306be4f] Lock freed to object EngineLock [exclusiveLocks= , sharedLocks= key: 00000002-0002-0002-0002-0000000001d9 value: POOL >] >2014-08-07 10:58:17,426 INFO [org.ovirt.engine.core.bll.MaintenanceVdsCommand] (org.ovirt.thread.pool-8-thread-10) [4306be4f] Running command: MaintenanceVdsCommand internal: true. Entities affected : ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS >2014-08-07 10:58:17,437 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-10) [4306be4f] Correlation ID: 4306be4f, Job ID: 03ca9fbb-39d3-4fd3-a560-9d66574f77e1, Call Stack: null, Custom Event ID: -1, Message: Host rack was switched to Maintenance mode by admin. >2014-08-07 10:58:17,971 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-46) Updated vds status from Preparing for Maintenance to Maintenance in database, vds = 17f7fe56-5f32-40ef-b098-611b84484e0b : rack >2014-08-07 10:58:17,994 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-8) Clearing cache of pool: 00000002-0002-0002-0002-0000000001d9 for problematic entities of VDS: rack. >2014-08-07 10:58:17,995 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-8) Removing vds [17f7fe56-5f32-40ef-b098-611b84484e0b] from the domain in maintenance cache >2014-08-07 10:58:17,996 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-8) Removing host(s) [17f7fe56-5f32-40ef-b098-611b84484e0b] from hosts unseen domain report cache >2014-08-07 10:58:18,001 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-46) START, DisconnectStoragePoolVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, vds_spm_id = 1), log id: 63f7f7c9 >2014-08-07 10:58:18,335 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-54) Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Active to Unknown, reason : no reporting hosts >2014-08-07 10:58:18,339 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-54) Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Active to Unknown, reason : no reporting hosts >2014-08-07 10:58:18,345 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-54) [43323ecc] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-08-07 10:58:18,354 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-54) [43323ecc] Correlation ID: 43323ecc, Call Stack: null, Custom Event ID: -1, Message: Invalid status on Data Center Default. Setting status to Non Responsive. >2014-08-07 10:58:22,097 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-46) FINISH, DisconnectStoragePoolVDSCommand, log id: 63f7f7c9 >2014-08-07 10:58:22,101 INFO [org.ovirt.engine.core.bll.storage.DisconnectHostFromStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-46) [54591d8f] Running command: DisconnectHostFromStoragePoolServersCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-08-07 10:58:22,126 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-46) [54591d8f] START, DisconnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, storageType = NFS, connectionList = [{ id: 05047c9a-152e-4ef2-b118-4a1753c1eae0, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35export, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 78657e38 >2014-08-07 10:58:22,319 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-46) [54591d8f] FINISH, DisconnectStorageServerVDSCommand, return: {5083f370-af09-457b-9b73-3440e8b65a07=0, 05047c9a-152e-4ef2-b118-4a1753c1eae0=0}, log id: 78657e38 >2014-08-07 10:58:41,520 INFO [org.ovirt.engine.core.bll.ChangeVDSClusterCommand] (ajp--127.0.0.1-8702-10) [5447dae2] Running command: ChangeVDSClusterCommand internal: false. Entities affected : ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS, ID: e14eab06-127a-4eaf-9220-ee1fd18c619e Type: VdsGroups >2014-08-07 10:58:41,643 INFO [org.ovirt.engine.core.vdsbroker.RemoveVdsVDSCommand] (ajp--127.0.0.1-8702-10) [5447dae2] START, RemoveVdsVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 79d1c4e3 >2014-08-07 10:58:41,644 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (ajp--127.0.0.1-8702-10) [5447dae2] vdsManager::disposing >2014-08-07 10:58:41,648 INFO [org.ovirt.engine.core.vdsbroker.RemoveVdsVDSCommand] (ajp--127.0.0.1-8702-10) [5447dae2] FINISH, RemoveVdsVDSCommand, log id: 79d1c4e3 >2014-08-07 10:58:41,652 INFO [org.ovirt.engine.core.vdsbroker.AddVdsVDSCommand] (ajp--127.0.0.1-8702-10) [5447dae2] START, AddVdsVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 1e9829cf >2014-08-07 10:58:41,653 INFO [org.ovirt.engine.core.vdsbroker.AddVdsVDSCommand] (ajp--127.0.0.1-8702-10) [5447dae2] AddVds - entered , starting logic to add VDS 17f7fe56-5f32-40ef-b098-611b84484e0b >2014-08-07 10:58:41,656 INFO [org.ovirt.engine.core.vdsbroker.AddVdsVDSCommand] (ajp--127.0.0.1-8702-10) [5447dae2] AddVds - VDS 17f7fe56-5f32-40ef-b098-611b84484e0b was added, will try to add it to the resource manager >2014-08-07 10:58:41,657 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (ajp--127.0.0.1-8702-10) [5447dae2] Entered VdsManager constructor >2014-08-07 10:58:41,689 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (ajp--127.0.0.1-8702-10) [5447dae2] Initialize vdsBroker (10.34.62.208,54,321) >2014-08-07 10:58:41,692 INFO [org.ovirt.engine.core.vdsbroker.ResourceManager] (ajp--127.0.0.1-8702-10) [5447dae2] VDS 17f7fe56-5f32-40ef-b098-611b84484e0b was added to the Resource Manager >2014-08-07 10:58:41,693 INFO [org.ovirt.engine.core.vdsbroker.AddVdsVDSCommand] (ajp--127.0.0.1-8702-10) [5447dae2] FINISH, AddVdsVDSCommand, log id: 1e9829cf >2014-08-07 10:58:41,701 INFO [org.ovirt.engine.core.bll.AddVdsSpmIdCommand] (ajp--127.0.0.1-8702-10) [5f80601f] Running command: AddVdsSpmIdCommand internal: true. Entities affected : ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS >2014-08-07 10:58:41,724 INFO [org.ovirt.engine.core.bll.AddVdsSpmIdCommand] (ajp--127.0.0.1-8702-10) [5f80601f] Lock freed to object EngineLock [exclusiveLocks= key: 29cf5221-8594-4e8b-afd3-bb0227304f01 value: REGISTER_VDS >, sharedLocks= ] >2014-08-07 10:58:41,755 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-10) [5f80601f] Correlation ID: 5447dae2, Call Stack: null, Custom Event ID: -1, Message: Host rack configuration was updated by admin. >2014-08-07 10:58:41,804 INFO [org.ovirt.engine.core.bll.network.host.PersistentSetupNetworksCommand] (org.ovirt.thread.pool-8-thread-31) [5b378ea] Running command: PersistentSetupNetworksCommand internal: true. Entities affected : ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS >2014-08-07 10:58:41,852 INFO [org.ovirt.engine.core.bll.network.host.SetupNetworksCommand] (org.ovirt.thread.pool-8-thread-31) [5b378ea] Running command: SetupNetworksCommand internal: true. Entities affected : ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS >2014-08-07 10:58:41,855 INFO [org.ovirt.engine.core.bll.network.host.SetupNetworksCommand] (org.ovirt.thread.pool-8-thread-31) [5b378ea] No changes were detected in setup networks for host rack (ID: 17f7fe56-5f32-40ef-b098-611b84484e0b) >2014-08-07 10:58:41,941 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-2) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-07 10:58:41,942 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-2) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-07 10:58:41,949 INFO [org.ovirt.engine.core.bll.UpdateVdsCommand] (ajp--127.0.0.1-8702-2) [4d6f003d] Running command: UpdateVdsCommand internal: false. Entities affected : ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS >2014-08-07 10:58:41,977 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-2) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-07 10:58:41,978 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-2) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-07 10:58:41,985 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-2) [4d6f003d] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host rack. >2014-08-07 10:58:42,005 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-2) [4d6f003d] Correlation ID: 4d6f003d, Call Stack: null, Custom Event ID: -1, Message: Host rack configuration was updated by admin. >2014-08-07 10:58:44,692 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-68) Initializing Host: rack >2014-08-07 10:58:45,398 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (ajp--127.0.0.1-8702-5) [7800e596] Lock Acquired to object EngineLock [exclusiveLocks= key: 17f7fe56-5f32-40ef-b098-611b84484e0b value: VDS >, sharedLocks= ] >2014-08-07 10:58:45,423 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (org.ovirt.thread.pool-8-thread-38) [7800e596] Running command: ActivateVdsCommand internal: false. Entities affected : ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS >2014-08-07 10:58:45,425 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (org.ovirt.thread.pool-8-thread-38) [7800e596] Before acquiring lock in order to prevent monitoring for host rack from data-center local >2014-08-07 10:58:45,426 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (org.ovirt.thread.pool-8-thread-38) [7800e596] Lock acquired, from now a monitoring of host will be skipped for host rack from data-center local >2014-08-07 10:58:45,436 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-8-thread-38) [7800e596] START, SetVdsStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 4cdc5d3b >2014-08-07 10:58:45,443 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-8-thread-38) [7800e596] FINISH, SetVdsStatusVDSCommand, log id: 4cdc5d3b >2014-08-07 10:58:45,452 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (org.ovirt.thread.pool-8-thread-38) [7800e596] Activate finished. Lock released. Monitoring can run now for host rack from data-center local >2014-08-07 10:58:45,458 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-38) [7800e596] Correlation ID: 7800e596, Job ID: 59ef2e2f-a80f-4cf7-ba48-c9c5dc27d587, Call Stack: null, Custom Event ID: -1, Message: Host rack was activated by admin. >2014-08-07 10:58:45,463 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (org.ovirt.thread.pool-8-thread-38) [7800e596] Lock freed to object EngineLock [exclusiveLocks= key: 17f7fe56-5f32-40ef-b098-611b84484e0b value: VDS >, sharedLocks= ] >2014-08-07 10:58:47,783 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-91) START, GetHardwareInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]), log id: 6a55d48c >2014-08-07 10:58:47,839 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-91) FINISH, GetHardwareInfoVDSCommand, log id: 6a55d48c >2014-08-07 10:58:47,867 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-91) [5fab0ca1] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS >2014-08-07 10:58:47,923 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-91) [5fab0ca1] START, GetHardwareInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]), log id: 663e95a8 >2014-08-07 10:58:47,970 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-91) [5fab0ca1] FINISH, GetHardwareInfoVDSCommand, log id: 663e95a8 >2014-08-07 10:58:48,037 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-91) [553563f7] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 29cf5221-8594-4e8b-afd3-bb0227304f01 Type: StoragePool >2014-08-07 10:58:48,048 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-91) [553563f7] START, SetMOMPolicyParametersVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 15d9180e >2014-08-07 10:58:48,141 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-91) [553563f7] FINISH, SetMOMPolicyParametersVDSCommand, log id: 15d9180e >2014-08-07 10:58:48,148 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-91) [553563f7] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host rack was set to Up. >2014-08-07 10:58:48,154 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-91) [553563f7] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host rack. >2014-08-07 10:58:48,182 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-91) [2899c446] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS >2014-08-07 11:09:23,517 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-7) Running command: LoginUserCommand internal: false. >2014-08-07 11:09:23,523 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-7) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-08-07 11:09:24,379 INFO [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp--127.0.0.1-8702-7) [a701299] Running command: LogoutBySessionCommand internal: false. >2014-08-07 11:09:24,388 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp--127.0.0.1-8702-7) [3ada5403] Running command: LogoutUserCommand internal: false. >2014-08-07 11:09:24,396 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-7) [3ada5403] Correlation ID: 3ada5403, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-08-07 11:09:24,532 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-6) Running command: LoginUserCommand internal: false. >2014-08-07 11:09:24,548 INFO [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp--127.0.0.1-8702-6) [5e8f34a1] Running command: LogoutBySessionCommand internal: false. >2014-08-07 11:09:24,553 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp--127.0.0.1-8702-6) [45264bb0] Running command: LogoutUserCommand internal: false. >2014-08-07 11:09:24,558 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) [45264bb0] Correlation ID: 45264bb0, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-08-07 11:09:45,579 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-9) Running command: LoginUserCommand internal: false. >2014-08-07 11:09:45,584 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-9) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-08-07 11:09:45,714 INFO [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp--127.0.0.1-8702-9) [6f689abc] Running command: LogoutBySessionCommand internal: false. >2014-08-07 11:09:45,721 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp--127.0.0.1-8702-9) [7900944a] Running command: LogoutUserCommand internal: false. >2014-08-07 11:09:45,727 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-9) [7900944a] Correlation ID: 7900944a, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-08-07 11:09:45,824 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-3) Running command: LoginUserCommand internal: false. >2014-08-07 11:09:45,871 INFO [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp--127.0.0.1-8702-3) [456e296a] Running command: LogoutBySessionCommand internal: false. >2014-08-07 11:09:45,879 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp--127.0.0.1-8702-3) [7e41eb29] Running command: LogoutUserCommand internal: false. >2014-08-07 11:09:45,883 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-3) [7e41eb29] Correlation ID: 7e41eb29, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-08-07 11:10:01,860 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-7) Running command: LoginUserCommand internal: false. >2014-08-07 11:10:01,864 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-7) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-08-07 11:10:02,663 INFO [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp--127.0.0.1-8702-7) [1827f148] Running command: LogoutBySessionCommand internal: false. >2014-08-07 11:10:02,667 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp--127.0.0.1-8702-7) [71147093] Running command: LogoutUserCommand internal: false. >2014-08-07 11:10:02,671 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-7) [71147093] Correlation ID: 71147093, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-08-07 11:10:15,016 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-6) Running command: LoginUserCommand internal: false. >2014-08-07 11:10:15,019 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-08-07 11:10:15,047 INFO [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp--127.0.0.1-8702-6) [746ec960] Running command: LogoutBySessionCommand internal: false. >2014-08-07 11:10:15,051 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp--127.0.0.1-8702-6) [658768b3] Running command: LogoutUserCommand internal: false. >2014-08-07 11:10:15,055 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) [658768b3] Correlation ID: 658768b3, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-08-07 11:10:22,514 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-9) Running command: LoginUserCommand internal: false. >2014-08-07 11:10:22,520 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-9) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-08-07 11:10:22,582 INFO [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp--127.0.0.1-8702-9) [52d114e3] Running command: LogoutBySessionCommand internal: false. >2014-08-07 11:10:22,588 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp--127.0.0.1-8702-9) [1853ef19] Running command: LogoutUserCommand internal: false. >2014-08-07 11:10:22,594 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-9) [1853ef19] Correlation ID: 1853ef19, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-08-07 11:11:07,493 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-8) Running command: LoginUserCommand internal: false. >2014-08-07 11:11:07,497 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-8) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-08-07 11:11:07,537 INFO [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp--127.0.0.1-8702-8) [2edb5230] Running command: LogoutBySessionCommand internal: false. >2014-08-07 11:11:07,546 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp--127.0.0.1-8702-8) [6dc84935] Running command: LogoutUserCommand internal: false. >2014-08-07 11:11:07,550 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-8) [6dc84935] Correlation ID: 6dc84935, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-08-07 11:11:39,905 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-7) Running command: LoginUserCommand internal: false. >2014-08-07 11:11:39,909 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-7) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-08-07 11:11:39,927 INFO [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp--127.0.0.1-8702-7) [519319d3] Running command: LogoutBySessionCommand internal: false. >2014-08-07 11:11:39,929 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp--127.0.0.1-8702-7) [4efbd73e] Running command: LogoutUserCommand internal: false. >2014-08-07 11:11:39,933 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-7) [4efbd73e] Correlation ID: 4efbd73e, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-08-07 11:12:43,923 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-6) Running command: LoginUserCommand internal: false. >2014-08-07 11:12:43,931 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-08-07 11:12:44,001 INFO [org.ovirt.engine.core.bll.storage.AddStorageServerConnectionCommand] (ajp--127.0.0.1-8702-6) [3c536aa6] Lock Acquired to object EngineLock [exclusiveLocks= key: /storage/sd value: STORAGE_CONNECTION >, sharedLocks= ] >2014-08-07 11:12:44,030 INFO [org.ovirt.engine.core.bll.storage.AddStorageServerConnectionCommand] (ajp--127.0.0.1-8702-6) [3c536aa6] Running command: AddStorageServerConnectionCommand internal: false. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2014-08-07 11:12:44,043 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (ajp--127.0.0.1-8702-6) [3c536aa6] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = LOCALFS, connectionList = [{ id: null, connection: /storage/sd, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 7afdab99 >2014-08-07 11:12:44,072 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (ajp--127.0.0.1-8702-6) [3c536aa6] FINISH, ConnectStorageServerVDSCommand, return: {00000000-0000-0000-0000-000000000000=469}, log id: 7afdab99 >2014-08-07 11:12:44,081 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) [3c536aa6] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: The error message for connection /storage/sd returned by VDSM was: Permission settings on the specified path do not allow access to the storage. >Verify permission settings on the specified storage path. >2014-08-07 11:12:44,096 ERROR [org.ovirt.engine.core.bll.storage.LOCALFSStorageHelper] (ajp--127.0.0.1-8702-6) [3c536aa6] The connection with details /storage/sd failed because of error code 469 and error message is: permission settings on the specified path do not allow access to the storage. >verify permission settings on the specified storage path. >2014-08-07 11:12:44,107 ERROR [org.ovirt.engine.core.bll.storage.AddStorageServerConnectionCommand] (ajp--127.0.0.1-8702-6) [3c536aa6] Command org.ovirt.engine.core.bll.storage.AddStorageServerConnectionCommand throw Vdc Bll exception. With error message VdcBLLException: StorageServerAccessPermissionError (Failed with error StorageServerAccessPermissionError and code 469) >2014-08-07 11:12:44,122 ERROR [org.ovirt.engine.core.bll.storage.AddStorageServerConnectionCommand] (ajp--127.0.0.1-8702-6) [3c536aa6] Transaction rolled-back for command: org.ovirt.engine.core.bll.storage.AddStorageServerConnectionCommand. >2014-08-07 11:12:44,134 INFO [org.ovirt.engine.core.bll.storage.AddStorageServerConnectionCommand] (ajp--127.0.0.1-8702-6) [3c536aa6] Lock freed to object EngineLock [exclusiveLocks= key: /storage/sd value: STORAGE_CONNECTION >, sharedLocks= ] >2014-08-07 11:12:44,143 WARN [org.ovirt.engine.core.utils.ErrorTranslatorImpl] (ajp--127.0.0.1-8702-6) Code MAC_ADDRESS_IS_IN_USE appears more than once in string table. >2014-08-07 11:12:44,157 ERROR [org.ovirt.engine.api.restapi.resource.AbstractBackendResource] (ajp--127.0.0.1-8702-6) Operation Failed: [Permission settings on the specified path do not allow access to the storage. >Verify permission settings on the specified storage path.] >2014-08-07 11:12:44,737 INFO [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp--127.0.0.1-8702-6) [2ec5bc22] Running command: LogoutBySessionCommand internal: false. >2014-08-07 11:12:44,777 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp--127.0.0.1-8702-6) [2b5631f5] Running command: LogoutUserCommand internal: false. >2014-08-07 11:12:44,785 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) [2b5631f5] Correlation ID: 2b5631f5, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-08-07 11:13:16,389 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-10) Running command: LoginUserCommand internal: false. >2014-08-07 11:13:16,393 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-10) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-08-07 11:13:16,412 INFO [org.ovirt.engine.core.bll.storage.AddStorageServerConnectionCommand] (ajp--127.0.0.1-8702-10) [5e67349b] Lock Acquired to object EngineLock [exclusiveLocks= key: /storage value: STORAGE_CONNECTION >, sharedLocks= ] >2014-08-07 11:13:16,426 INFO [org.ovirt.engine.core.bll.storage.AddStorageServerConnectionCommand] (ajp--127.0.0.1-8702-10) [5e67349b] Running command: AddStorageServerConnectionCommand internal: false. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2014-08-07 11:13:16,433 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (ajp--127.0.0.1-8702-10) [5e67349b] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = LOCALFS, connectionList = [{ id: null, connection: /storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 3a4b8c71 >2014-08-07 11:13:16,467 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (ajp--127.0.0.1-8702-10) [5e67349b] FINISH, ConnectStorageServerVDSCommand, return: {00000000-0000-0000-0000-000000000000=100}, log id: 3a4b8c71 >2014-08-07 11:13:16,475 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-10) [5e67349b] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: The error message for connection /storage returned by VDSM was: General Exception >2014-08-07 11:13:16,483 ERROR [org.ovirt.engine.core.bll.storage.LOCALFSStorageHelper] (ajp--127.0.0.1-8702-10) [5e67349b] The connection with details /storage failed because of error code 100 and error message is: general exception >2014-08-07 11:13:16,488 ERROR [org.ovirt.engine.core.bll.storage.AddStorageServerConnectionCommand] (ajp--127.0.0.1-8702-10) [5e67349b] Command org.ovirt.engine.core.bll.storage.AddStorageServerConnectionCommand throw Vdc Bll exception. With error message VdcBLLException: GeneralException (Failed with error GeneralException and code 100) >2014-08-07 11:13:16,495 ERROR [org.ovirt.engine.core.bll.storage.AddStorageServerConnectionCommand] (ajp--127.0.0.1-8702-10) [5e67349b] Transaction rolled-back for command: org.ovirt.engine.core.bll.storage.AddStorageServerConnectionCommand. >2014-08-07 11:13:16,503 INFO [org.ovirt.engine.core.bll.storage.AddStorageServerConnectionCommand] (ajp--127.0.0.1-8702-10) [5e67349b] Lock freed to object EngineLock [exclusiveLocks= key: /storage value: STORAGE_CONNECTION >, sharedLocks= ] >2014-08-07 11:13:16,508 ERROR [org.ovirt.engine.api.restapi.resource.AbstractBackendResource] (ajp--127.0.0.1-8702-10) Operation Failed: [General Exception] >2014-08-07 11:13:16,530 INFO [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp--127.0.0.1-8702-10) [23db14df] Running command: LogoutBySessionCommand internal: false. >2014-08-07 11:13:16,535 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp--127.0.0.1-8702-10) [6a7821b4] Running command: LogoutUserCommand internal: false. >2014-08-07 11:13:16,541 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-10) [6a7821b4] Correlation ID: 6a7821b4, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-08-07 11:13:30,423 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-8) Running command: LoginUserCommand internal: false. >2014-08-07 11:13:30,438 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-8) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-08-07 11:13:30,473 INFO [org.ovirt.engine.core.bll.storage.AddStorageServerConnectionCommand] (ajp--127.0.0.1-8702-8) [7d3ffe71] Lock Acquired to object EngineLock [exclusiveLocks= key: /storage/sd value: STORAGE_CONNECTION >, sharedLocks= ] >2014-08-07 11:13:30,492 INFO [org.ovirt.engine.core.bll.storage.AddStorageServerConnectionCommand] (ajp--127.0.0.1-8702-8) [7d3ffe71] Running command: AddStorageServerConnectionCommand internal: false. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2014-08-07 11:13:30,502 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (ajp--127.0.0.1-8702-8) [7d3ffe71] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = LOCALFS, connectionList = [{ id: null, connection: /storage/sd, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 324196b4 >2014-08-07 11:13:30,522 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (ajp--127.0.0.1-8702-8) [7d3ffe71] FINISH, ConnectStorageServerVDSCommand, return: {00000000-0000-0000-0000-000000000000=469}, log id: 324196b4 >2014-08-07 11:13:30,530 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-8) [7d3ffe71] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: The error message for connection /storage/sd returned by VDSM was: Permission settings on the specified path do not allow access to the storage. >Verify permission settings on the specified storage path. >2014-08-07 11:13:30,544 ERROR [org.ovirt.engine.core.bll.storage.LOCALFSStorageHelper] (ajp--127.0.0.1-8702-8) [7d3ffe71] The connection with details /storage/sd failed because of error code 469 and error message is: permission settings on the specified path do not allow access to the storage. >verify permission settings on the specified storage path. >2014-08-07 11:13:30,555 ERROR [org.ovirt.engine.core.bll.storage.AddStorageServerConnectionCommand] (ajp--127.0.0.1-8702-8) [7d3ffe71] Command org.ovirt.engine.core.bll.storage.AddStorageServerConnectionCommand throw Vdc Bll exception. With error message VdcBLLException: StorageServerAccessPermissionError (Failed with error StorageServerAccessPermissionError and code 469) >2014-08-07 11:13:30,568 ERROR [org.ovirt.engine.core.bll.storage.AddStorageServerConnectionCommand] (ajp--127.0.0.1-8702-8) [7d3ffe71] Transaction rolled-back for command: org.ovirt.engine.core.bll.storage.AddStorageServerConnectionCommand. >2014-08-07 11:13:30,579 INFO [org.ovirt.engine.core.bll.storage.AddStorageServerConnectionCommand] (ajp--127.0.0.1-8702-8) [7d3ffe71] Lock freed to object EngineLock [exclusiveLocks= key: /storage/sd value: STORAGE_CONNECTION >, sharedLocks= ] >2014-08-07 11:13:30,587 ERROR [org.ovirt.engine.api.restapi.resource.AbstractBackendResource] (ajp--127.0.0.1-8702-8) Operation Failed: [Permission settings on the specified path do not allow access to the storage. >Verify permission settings on the specified storage path.] >2014-08-07 11:13:30,631 INFO [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp--127.0.0.1-8702-8) [2d7ea28a] Running command: LogoutBySessionCommand internal: false. >2014-08-07 11:13:30,639 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp--127.0.0.1-8702-8) [bd799c6] Running command: LogoutUserCommand internal: false. >2014-08-07 11:13:30,647 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-8) [bd799c6] Correlation ID: bd799c6, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-08-07 11:13:55,419 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-7) Running command: LoginUserCommand internal: false. >2014-08-07 11:13:55,427 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-7) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-08-07 11:13:55,453 INFO [org.ovirt.engine.core.bll.storage.AddStorageServerConnectionCommand] (ajp--127.0.0.1-8702-7) [5663f6d7] Lock Acquired to object EngineLock [exclusiveLocks= key: /storage/sd value: STORAGE_CONNECTION >, sharedLocks= ] >2014-08-07 11:13:55,468 INFO [org.ovirt.engine.core.bll.storage.AddStorageServerConnectionCommand] (ajp--127.0.0.1-8702-7) [5663f6d7] Running command: AddStorageServerConnectionCommand internal: false. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2014-08-07 11:13:55,473 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (ajp--127.0.0.1-8702-7) [5663f6d7] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = LOCALFS, connectionList = [{ id: null, connection: /storage/sd, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 5d426297 >2014-08-07 11:13:55,489 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (ajp--127.0.0.1-8702-7) [5663f6d7] FINISH, ConnectStorageServerVDSCommand, return: {00000000-0000-0000-0000-000000000000=469}, log id: 5d426297 >2014-08-07 11:13:55,497 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-7) [5663f6d7] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: The error message for connection /storage/sd returned by VDSM was: Permission settings on the specified path do not allow access to the storage. >Verify permission settings on the specified storage path. >2014-08-07 11:13:55,503 ERROR [org.ovirt.engine.core.bll.storage.LOCALFSStorageHelper] (ajp--127.0.0.1-8702-7) [5663f6d7] The connection with details /storage/sd failed because of error code 469 and error message is: permission settings on the specified path do not allow access to the storage. >verify permission settings on the specified storage path. >2014-08-07 11:13:55,509 ERROR [org.ovirt.engine.core.bll.storage.AddStorageServerConnectionCommand] (ajp--127.0.0.1-8702-7) [5663f6d7] Command org.ovirt.engine.core.bll.storage.AddStorageServerConnectionCommand throw Vdc Bll exception. With error message VdcBLLException: StorageServerAccessPermissionError (Failed with error StorageServerAccessPermissionError and code 469) >2014-08-07 11:13:55,516 ERROR [org.ovirt.engine.core.bll.storage.AddStorageServerConnectionCommand] (ajp--127.0.0.1-8702-7) [5663f6d7] Transaction rolled-back for command: org.ovirt.engine.core.bll.storage.AddStorageServerConnectionCommand. >2014-08-07 11:13:55,524 INFO [org.ovirt.engine.core.bll.storage.AddStorageServerConnectionCommand] (ajp--127.0.0.1-8702-7) [5663f6d7] Lock freed to object EngineLock [exclusiveLocks= key: /storage/sd value: STORAGE_CONNECTION >, sharedLocks= ] >2014-08-07 11:13:55,528 ERROR [org.ovirt.engine.api.restapi.resource.AbstractBackendResource] (ajp--127.0.0.1-8702-7) Operation Failed: [Permission settings on the specified path do not allow access to the storage. >Verify permission settings on the specified storage path.] >2014-08-07 11:13:55,550 INFO [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp--127.0.0.1-8702-7) [248841de] Running command: LogoutBySessionCommand internal: false. >2014-08-07 11:13:55,555 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp--127.0.0.1-8702-7) [668eb07d] Running command: LogoutUserCommand internal: false. >2014-08-07 11:13:55,562 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-7) [668eb07d] Correlation ID: 668eb07d, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-08-07 11:14:04,931 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-1) Running command: LoginUserCommand internal: false. >2014-08-07 11:14:04,938 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-1) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-08-07 11:14:04,998 INFO [org.ovirt.engine.core.bll.storage.AddStorageServerConnectionCommand] (ajp--127.0.0.1-8702-1) [7ed23bd4] Lock Acquired to object EngineLock [exclusiveLocks= key: /storage value: STORAGE_CONNECTION >, sharedLocks= ] >2014-08-07 11:14:05,023 INFO [org.ovirt.engine.core.bll.storage.AddStorageServerConnectionCommand] (ajp--127.0.0.1-8702-1) [7ed23bd4] Running command: AddStorageServerConnectionCommand internal: false. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2014-08-07 11:14:05,040 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (ajp--127.0.0.1-8702-1) [7ed23bd4] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = LOCALFS, connectionList = [{ id: null, connection: /storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 6ab0955e >2014-08-07 11:14:05,075 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (ajp--127.0.0.1-8702-1) [7ed23bd4] FINISH, ConnectStorageServerVDSCommand, return: {00000000-0000-0000-0000-000000000000=0}, log id: 6ab0955e >2014-08-07 11:14:05,093 INFO [org.ovirt.engine.core.bll.storage.AddStorageServerConnectionCommand] (ajp--127.0.0.1-8702-1) [7ed23bd4] Lock freed to object EngineLock [exclusiveLocks= key: /storage value: STORAGE_CONNECTION >, sharedLocks= ] >2014-08-07 11:14:05,151 INFO [org.ovirt.engine.core.bll.storage.AddLocalStorageDomainCommand] (ajp--127.0.0.1-8702-1) [5ca7e5d1] Running command: AddLocalStorageDomainCommand internal: false. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2014-08-07 11:14:05,186 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (ajp--127.0.0.1-8702-1) [5ca7e5d1] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = LOCALFS, connectionList = [{ id: eb6f1d12-a223-4dc0-abc3-5fa351718750, connection: /storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 144338b2 >2014-08-07 11:14:05,207 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (ajp--127.0.0.1-8702-1) [5ca7e5d1] FINISH, ConnectStorageServerVDSCommand, return: {eb6f1d12-a223-4dc0-abc3-5fa351718750=0}, log id: 144338b2 >2014-08-07 11:14:05,211 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateStorageDomainVDSCommand] (ajp--127.0.0.1-8702-1) [5ca7e5d1] START, CreateStorageDomainVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storageDomain=org.ovirt.engine.core.common.businessentities.StorageDomainStatic@84831e5f, args=/storage), log id: 393be476 >2014-08-07 11:15:05,690 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateStorageDomainVDSCommand] (ajp--127.0.0.1-8702-1) [5ca7e5d1] FINISH, CreateStorageDomainVDSCommand, log id: 393be476 >2014-08-07 11:15:05,696 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetStorageDomainStatsVDSCommand] (ajp--127.0.0.1-8702-1) [5ca7e5d1] START, GetStorageDomainStatsVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storageDomainId=37f3c12d-f0b0-4916-8e01-9a290cd82ed0), log id: 11c193a7 >2014-08-07 11:15:05,754 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetStorageDomainStatsVDSCommand] (ajp--127.0.0.1-8702-1) [5ca7e5d1] FINISH, GetStorageDomainStatsVDSCommand, return: org.ovirt.engine.core.common.businessentities.StorageDomain@6063117b, log id: 11c193a7 >2014-08-07 11:15:05,818 INFO [org.ovirt.engine.core.bll.storage.AttachStorageDomainToPoolCommand] (ajp--127.0.0.1-8702-1) [8735d5] Running command: AttachStorageDomainToPoolCommand internal: true. Entities affected : ID: 37f3c12d-f0b0-4916-8e01-9a290cd82ed0 Type: Storage >2014-08-07 11:15:05,832 INFO [org.ovirt.engine.core.bll.storage.AddStoragePoolWithStoragesCommand] (ajp--127.0.0.1-8702-1) [67d68dad] Lock Acquired to object EngineLock [exclusiveLocks= key: 29cf5221-8594-4e8b-afd3-bb0227304f01 value: POOL >, sharedLocks= ] >2014-08-07 11:15:05,863 INFO [org.ovirt.engine.core.bll.storage.AddStoragePoolWithStoragesCommand] (ajp--127.0.0.1-8702-1) [67d68dad] Running command: AddStoragePoolWithStoragesCommand internal: true. Entities affected : ID: 29cf5221-8594-4e8b-afd3-bb0227304f01 Type: StoragePool >2014-08-07 11:15:06,005 INFO [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (ajp--127.0.0.1-8702-1) [4ba10f4] Running command: ConnectStorageToVdsCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2014-08-07 11:15:06,013 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (ajp--127.0.0.1-8702-1) [4ba10f4] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = LOCALFS, connectionList = [{ id: eb6f1d12-a223-4dc0-abc3-5fa351718750, connection: /storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 79d3b6ea >2014-08-07 11:15:06,039 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (ajp--127.0.0.1-8702-1) [4ba10f4] FINISH, ConnectStorageServerVDSCommand, return: {eb6f1d12-a223-4dc0-abc3-5fa351718750=0}, log id: 79d3b6ea >2014-08-07 11:15:06,051 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateStoragePoolVDSCommand] (ajp--127.0.0.1-8702-1) [4ba10f4] START, CreateStoragePoolVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId=29cf5221-8594-4e8b-afd3-bb0227304f01, storagePoolName=local, masterDomainId=37f3c12d-f0b0-4916-8e01-9a290cd82ed0, domainsIdList=[37f3c12d-f0b0-4916-8e01-9a290cd82ed0], masterVersion=1), log id: 261cb48c >2014-08-07 11:16:06,427 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateStoragePoolVDSCommand] (ajp--127.0.0.1-8702-1) [4ba10f4] FINISH, CreateStoragePoolVDSCommand, log id: 261cb48c >2014-08-07 11:16:06,435 INFO [org.ovirt.engine.core.bll.storage.AddStoragePoolWithStoragesCommand] (ajp--127.0.0.1-8702-1) [4ba10f4] Lock freed to object EngineLock [exclusiveLocks= key: 29cf5221-8594-4e8b-afd3-bb0227304f01 value: POOL >, sharedLocks= ] >2014-08-07 11:16:06,440 INFO [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (ajp--127.0.0.1-8702-1) [4b64550f] Lock Acquired to object EngineLock [exclusiveLocks= key: 37f3c12d-f0b0-4916-8e01-9a290cd82ed0 value: STORAGE >, sharedLocks= ] >2014-08-07 11:16:06,450 INFO [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (ajp--127.0.0.1-8702-1) [4b64550f] Running command: ActivateStorageDomainCommand internal: true. Entities affected : ID: 37f3c12d-f0b0-4916-8e01-9a290cd82ed0 Type: Storage >2014-08-07 11:16:06,462 INFO [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (ajp--127.0.0.1-8702-1) [4b64550f] Lock freed to object EngineLock [exclusiveLocks= key: 37f3c12d-f0b0-4916-8e01-9a290cd82ed0 value: STORAGE >, sharedLocks= ] >2014-08-07 11:16:06,469 INFO [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (ajp--127.0.0.1-8702-1) [4b64550f] ActivateStorage Domain. Before Connect all hosts to pool. Time:8/7/14 11:16 AM >2014-08-07 11:16:06,477 INFO [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-8-thread-40) [4f05bc8a] Running command: ConnectStorageToVdsCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2014-08-07 11:16:06,482 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-40) [4f05bc8a] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = LOCALFS, connectionList = [{ id: eb6f1d12-a223-4dc0-abc3-5fa351718750, connection: /storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 233421ba >2014-08-07 11:16:06,500 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-40) [4f05bc8a] FINISH, ConnectStorageServerVDSCommand, return: {eb6f1d12-a223-4dc0-abc3-5fa351718750=0}, log id: 233421ba >2014-08-07 11:16:06,506 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ActivateStorageDomainVDSCommand] (ajp--127.0.0.1-8702-1) [4b64550f] START, ActivateStorageDomainVDSCommand( storagePoolId = 29cf5221-8594-4e8b-afd3-bb0227304f01, ignoreFailoverLimit = false, storageDomainId = 37f3c12d-f0b0-4916-8e01-9a290cd82ed0), log id: 44f218e1 >2014-08-07 11:16:06,518 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (ajp--127.0.0.1-8702-1) [4cca6b4f] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 29cf5221-8594-4e8b-afd3-bb0227304f01 Type: StoragePool >2014-08-07 11:16:06,541 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-1) [4cca6b4f] Correlation ID: 4cca6b4f, Call Stack: null, Custom Event ID: -1, Message: Data Center is being initialized, please wait for initialization to complete. >2014-08-07 11:16:06,558 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (ajp--127.0.0.1-8702-1) [4cca6b4f] hostFromVds::selectedVds - rack, spmStatus Unknown_Pool, storage pool local >2014-08-07 11:16:06,575 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (ajp--127.0.0.1-8702-1) [4cca6b4f] START, ConnectStoragePoolVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 29cf5221-8594-4e8b-afd3-bb0227304f01, masterVersion = 1), log id: 66cc42e7 >2014-08-07 11:17:06,822 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (ajp--127.0.0.1-8702-1) [4cca6b4f] FINISH, ConnectStoragePoolVDSCommand, log id: 66cc42e7 >2014-08-07 11:17:06,868 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (ajp--127.0.0.1-8702-1) [4cca6b4f] hostFromVds::Connected host to pool - selectedVds - rack, spmStatus Free, storage pool local >2014-08-07 11:17:06,874 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (ajp--127.0.0.1-8702-1) [4cca6b4f] starting spm on vds rack, storage pool local, prevId -1, LVER -1 >2014-08-07 11:17:06,883 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (ajp--127.0.0.1-8702-1) [4cca6b4f] START, SpmStartVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 29cf5221-8594-4e8b-afd3-bb0227304f01, prevId=-1, prevLVER=-1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 10aa41bf >2014-08-07 11:17:06,943 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (ajp--127.0.0.1-8702-1) [4cca6b4f] spmStart polling started: taskId = 7c4cd73d-477c-467f-a023-2e127d2de394 >2014-08-07 11:17:07,952 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (ajp--127.0.0.1-8702-1) [4cca6b4f] spmStart polling ended: taskId = 7c4cd73d-477c-467f-a023-2e127d2de394 task status = finished >2014-08-07 11:17:07,999 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (ajp--127.0.0.1-8702-1) [4cca6b4f] spmStart polling ended, spm status: SPM >2014-08-07 11:17:08,003 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (ajp--127.0.0.1-8702-1) [4cca6b4f] START, HSMClearTaskVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, taskId=7c4cd73d-477c-467f-a023-2e127d2de394), log id: afcb23c >2014-08-07 11:17:08,051 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (ajp--127.0.0.1-8702-1) [4cca6b4f] FINISH, HSMClearTaskVDSCommand, log id: afcb23c >2014-08-07 11:17:08,057 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (ajp--127.0.0.1-8702-1) [4cca6b4f] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@474618fa, log id: 10aa41bf >2014-08-07 11:17:08,068 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (ajp--127.0.0.1-8702-1) [4cca6b4f] Initialize Irs proxy from vds: 10.34.62.208 >2014-08-07 11:17:08,081 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-1) [4cca6b4f] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208). >2014-08-07 11:17:08,149 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ActivateStorageDomainVDSCommand] (ajp--127.0.0.1-8702-1) [4cca6b4f] FINISH, ActivateStorageDomainVDSCommand, log id: 44f218e1 >2014-08-07 11:17:08,152 INFO [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (ajp--127.0.0.1-8702-1) [4cca6b4f] ActivateStorage Domain. After Connect all hosts to pool. Time:8/7/14 11:17 AM >2014-08-07 11:17:08,171 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-33) START, ConnectStoragePoolVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 29cf5221-8594-4e8b-afd3-bb0227304f01, masterVersion = 1), log id: 5b07c499 >2014-08-07 11:18:08,388 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-33) FINISH, ConnectStoragePoolVDSCommand, log id: 5b07c499 >2014-08-07 11:18:08,390 INFO [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (ajp--127.0.0.1-8702-1) [4cca6b4f] ActivateStorage Domain. After change storage pool status in vds. Time:8/7/14 11:18 AM >2014-08-07 11:18:08,401 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-1) [4cca6b4f] Correlation ID: 4b64550f, Call Stack: null, Custom Event ID: -1, Message: Storage Domain local_sd (Data Center local) was activated by <UNKNOWN> >2014-08-07 11:18:08,421 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-1) [4cca6b4f] Correlation ID: 67d68dad, Call Stack: null, Custom Event ID: -1, Message: Storage Domains were attached to Data Center local by <UNKNOWN> >2014-08-07 11:18:08,437 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-1) [4cca6b4f] Correlation ID: 8735d5, Call Stack: null, Custom Event ID: -1, Message: Storage Domain local_sd was attached to Data Center local by <UNKNOWN> >2014-08-07 11:18:08,466 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-1) [4cca6b4f] Correlation ID: 5ca7e5d1, Job ID: 61d312f1-ada1-4d3b-9ee4-4f1bf0f8b6f9, Call Stack: null, Custom Event ID: -1, Message: Storage Domain local_sd was added by admin >2014-08-07 11:18:09,281 INFO [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp--127.0.0.1-8702-1) [47a99ab4] Running command: LogoutBySessionCommand internal: false. >2014-08-07 11:18:09,286 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp--127.0.0.1-8702-1) [4bcd6fb9] Running command: LogoutUserCommand internal: false. >2014-08-07 11:18:09,291 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-1) [4bcd6fb9] Correlation ID: 4bcd6fb9, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-08-07 11:18:19,370 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-7) Running command: LoginUserCommand internal: false. >2014-08-07 11:18:19,374 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-7) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-08-07 11:18:19,397 INFO [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp--127.0.0.1-8702-7) [1dc9b31c] Running command: LogoutBySessionCommand internal: false. >2014-08-07 11:18:19,408 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp--127.0.0.1-8702-7) [2c7b6724] Running command: LogoutUserCommand internal: false. >2014-08-07 11:18:19,419 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-7) [2c7b6724] Correlation ID: 2c7b6724, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-08-07 11:20:32,722 INFO [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (ajp--127.0.0.1-8702-7) [6e2c45bf] Lock Acquired to object EngineLock [exclusiveLocks= key: 37f3c12d-f0b0-4916-8e01-9a290cd82ed0 value: STORAGE >key: 29cf5221-8594-4e8b-afd3-bb0227304f01 value: POOL >, sharedLocks= ] >2014-08-07 11:20:32,812 INFO [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-21) [6e2c45bf] Running command: DeactivateStorageDomainCommand internal: false. Entities affected : ID: 37f3c12d-f0b0-4916-8e01-9a290cd82ed0 Type: Storage >2014-08-07 11:20:32,832 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.DeactivateStorageDomainVDSCommand] (org.ovirt.thread.pool-8-thread-21) [6e2c45bf] START, DeactivateStorageDomainVDSCommand( storagePoolId = 29cf5221-8594-4e8b-afd3-bb0227304f01, ignoreFailoverLimit = false, storageDomainId = 37f3c12d-f0b0-4916-8e01-9a290cd82ed0, masterDomainId = 00000000-0000-0000-0000-000000000000, masterVersion = 1), log id: 4561ad89 >2014-08-07 11:20:32,842 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.DeactivateStorageDomainVDSCommand] (org.ovirt.thread.pool-8-thread-21) [6e2c45bf] FINISH, DeactivateStorageDomainVDSCommand, log id: 4561ad89 >2014-08-07 11:20:32,844 INFO [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-21) [6e2c45bf] Lock freed to object EngineLock [exclusiveLocks= key: 37f3c12d-f0b0-4916-8e01-9a290cd82ed0 value: STORAGE >key: 29cf5221-8594-4e8b-afd3-bb0227304f01 value: POOL >, sharedLocks= ] >2014-08-07 11:20:32,846 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SpmStopOnIrsVDSCommand] (org.ovirt.thread.pool-8-thread-21) [6e2c45bf] START, SpmStopOnIrsVDSCommand( storagePoolId = 29cf5221-8594-4e8b-afd3-bb0227304f01, ignoreFailoverLimit = false), log id: 5990381 >2014-08-07 11:20:32,848 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-21) [6e2c45bf] START, ResetIrsVDSCommand( storagePoolId = 29cf5221-8594-4e8b-afd3-bb0227304f01, ignoreFailoverLimit = false, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, ignoreStopFailed = false), log id: 72806f4b >2014-08-07 11:20:32,852 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-21) [6e2c45bf] START, SpmStopVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 29cf5221-8594-4e8b-afd3-bb0227304f01), log id: 6b22d588 >2014-08-07 11:20:32,859 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-21) [6e2c45bf] SpmStopVDSCommand::Stopping SPM on vds rack, pool id 29cf5221-8594-4e8b-afd3-bb0227304f01 >2014-08-07 11:20:32,907 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-21) [6e2c45bf] FINISH, SpmStopVDSCommand, log id: 6b22d588 >2014-08-07 11:20:32,913 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-21) [6e2c45bf] FINISH, ResetIrsVDSCommand, log id: 72806f4b >2014-08-07 11:20:32,914 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SpmStopOnIrsVDSCommand] (org.ovirt.thread.pool-8-thread-21) [6e2c45bf] FINISH, SpmStopOnIrsVDSCommand, log id: 5990381 >2014-08-07 11:20:32,920 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-21) [6e2c45bf] START, DisconnectStoragePoolVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 29cf5221-8594-4e8b-afd3-bb0227304f01, vds_spm_id = 1), log id: 4113d88d >2014-08-07 11:20:32,972 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-21) [6e2c45bf] FINISH, DisconnectStoragePoolVDSCommand, log id: 4113d88d >2014-08-07 11:20:32,975 INFO [org.ovirt.engine.core.bll.storage.DisconnectStoragePoolAsyncOperation] (org.ovirt.thread.pool-8-thread-50) [6e2c45bf] Disconnect storage pool treatment vds: rack,pool local >2014-08-07 11:20:32,976 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-50) [6e2c45bf] START, DisconnectStoragePoolVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 29cf5221-8594-4e8b-afd3-bb0227304f01, vds_spm_id = 1), log id: 5b306cb2 >2014-08-07 11:20:33,022 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-50) [6e2c45bf] FINISH, DisconnectStoragePoolVDSCommand, log id: 5b306cb2 >2014-08-07 11:20:33,029 INFO [org.ovirt.engine.core.bll.storage.AfterDeactivateSingleAsyncOperation] (org.ovirt.thread.pool-8-thread-4) After deactivate treatment vds: rack,pool local >2014-08-07 11:20:33,034 INFO [org.ovirt.engine.core.bll.storage.DisconnectStorageServerConnectionCommand] (org.ovirt.thread.pool-8-thread-4) [59bc1f82] Running command: DisconnectStorageServerConnectionCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2014-08-07 11:20:33,038 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-4) [59bc1f82] START, DisconnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = LOCALFS, connectionList = [{ id: eb6f1d12-a223-4dc0-abc3-5fa351718750, connection: /storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 567f46a6 >2014-08-07 11:20:33,188 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-4) [59bc1f82] FINISH, DisconnectStorageServerVDSCommand, return: {eb6f1d12-a223-4dc0-abc3-5fa351718750=0}, log id: 567f46a6 >2014-08-07 11:20:33,192 INFO [org.ovirt.engine.core.bll.storage.DisconnectStorageServerConnectionCommand] (org.ovirt.thread.pool-8-thread-21) [26d1eabb] Running command: DisconnectStorageServerConnectionCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2014-08-07 11:20:33,195 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-21) [26d1eabb] START, DisconnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = LOCALFS, connectionList = [{ id: eb6f1d12-a223-4dc0-abc3-5fa351718750, connection: /storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 7aa7c118 >2014-08-07 11:20:33,340 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-21) [26d1eabb] FINISH, DisconnectStorageServerVDSCommand, return: {eb6f1d12-a223-4dc0-abc3-5fa351718750=0}, log id: 7aa7c118 >2014-08-07 11:20:33,357 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-21) [26d1eabb] Correlation ID: 6e2c45bf, Job ID: 826fd762-3cfd-4894-a497-7c145b346c2a, Call Stack: null, Custom Event ID: -1, Message: Storage Domain local_sd (Data Center local) was deactivated by admin >2014-08-07 11:30:54,997 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-4) Principal admin::internal synchronized >2014-08-07 11:30:55,806 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-91) [2899c446] Attempting to update VMs/Templates Ovf. >2014-08-07 12:17:31,446 INFO [org.ovirt.engine.core.bll.storage.RemoveStorageDomainCommand] (ajp--127.0.0.1-8702-3) [3fcc0f5] Lock Acquired to object EngineLock [exclusiveLocks= key: 37f3c12d-f0b0-4916-8e01-9a290cd82ed0 value: STORAGE >, sharedLocks= ] >2014-08-07 12:17:31,458 WARN [org.ovirt.engine.core.bll.storage.RemoveStorageDomainCommand] (ajp--127.0.0.1-8702-3) [3fcc0f5] CanDoAction of action RemoveStorageDomain failed. Reasons:VAR__TYPE__STORAGE__DOMAIN,VAR__ACTION__REMOVE,ERROR_CANNOT_DETACH_LAST_STORAGE_DOMAIN >2014-08-07 12:17:31,459 INFO [org.ovirt.engine.core.bll.storage.RemoveStorageDomainCommand] (ajp--127.0.0.1-8702-3) [3fcc0f5] Lock freed to object EngineLock [exclusiveLocks= key: 37f3c12d-f0b0-4916-8e01-9a290cd82ed0 value: STORAGE >, sharedLocks= ] >2014-08-07 12:17:45,475 INFO [org.ovirt.engine.core.bll.MaintenanceNumberOfVdssCommand] (org.ovirt.thread.pool-8-thread-12) [3900667c] Running command: MaintenanceNumberOfVdssCommand internal: false. Entities affected : ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS >2014-08-07 12:17:45,480 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-8-thread-12) [3900667c] START, SetVdsStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, status=PreparingForMaintenance, nonOperationalReason=NONE, stopSpmFailureLogged=true), log id: 424f728f >2014-08-07 12:17:45,486 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-8-thread-12) [3900667c] FINISH, SetVdsStatusVDSCommand, log id: 424f728f >2014-08-07 12:17:45,567 INFO [org.ovirt.engine.core.bll.MaintenanceVdsCommand] (org.ovirt.thread.pool-8-thread-12) [3900667c] Running command: MaintenanceVdsCommand internal: true. Entities affected : ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS >2014-08-07 12:17:45,576 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-12) [3900667c] Correlation ID: 3900667c, Job ID: 1c7a7ce7-9366-4a76-bfa1-1c3cc8080fa6, Call Stack: null, Custom Event ID: -1, Message: Host rack was switched to Maintenance mode by admin. >2014-08-07 12:17:45,845 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-31) Updated vds status from Preparing for Maintenance to Maintenance in database, vds = 17f7fe56-5f32-40ef-b098-611b84484e0b : rack >2014-08-07 12:17:45,868 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-32) Clearing cache of pool: 29cf5221-8594-4e8b-afd3-bb0227304f01 for problematic entities of VDS: rack. >2014-08-07 12:17:45,870 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-32) Removing vds [17f7fe56-5f32-40ef-b098-611b84484e0b] from the domain in maintenance cache >2014-08-07 12:17:45,871 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-32) Removing host(s) [17f7fe56-5f32-40ef-b098-611b84484e0b] from hosts unseen domain report cache >2014-08-07 12:17:45,875 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-31) START, DisconnectStoragePoolVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 29cf5221-8594-4e8b-afd3-bb0227304f01, vds_spm_id = 1), log id: 69413136 >2014-08-07 12:17:45,921 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-31) FINISH, DisconnectStoragePoolVDSCommand, log id: 69413136 >2014-08-07 12:17:45,926 INFO [org.ovirt.engine.core.bll.storage.DisconnectHostFromStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-31) [8b21b6c] Running command: DisconnectHostFromStoragePoolServersCommand internal: true. Entities affected : ID: 29cf5221-8594-4e8b-afd3-bb0227304f01 Type: StoragePool >2014-08-07 12:18:01,090 INFO [org.ovirt.engine.core.bll.ChangeVDSClusterCommand] (ajp--127.0.0.1-8702-2) [5d35448e] Running command: ChangeVDSClusterCommand internal: false. Entities affected : ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS, ID: 00000001-0001-0001-0001-0000000002c2 Type: VdsGroups >2014-08-07 12:18:01,100 INFO [org.ovirt.engine.core.vdsbroker.RemoveVdsVDSCommand] (ajp--127.0.0.1-8702-2) [5d35448e] START, RemoveVdsVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 30a79b77 >2014-08-07 12:18:01,101 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (ajp--127.0.0.1-8702-2) [5d35448e] vdsManager::disposing >2014-08-07 12:18:01,102 INFO [org.ovirt.engine.core.vdsbroker.RemoveVdsVDSCommand] (ajp--127.0.0.1-8702-2) [5d35448e] FINISH, RemoveVdsVDSCommand, log id: 30a79b77 >2014-08-07 12:18:01,105 INFO [org.ovirt.engine.core.vdsbroker.AddVdsVDSCommand] (ajp--127.0.0.1-8702-2) [5d35448e] START, AddVdsVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 4e3cd999 >2014-08-07 12:18:01,106 INFO [org.ovirt.engine.core.vdsbroker.AddVdsVDSCommand] (ajp--127.0.0.1-8702-2) [5d35448e] AddVds - entered , starting logic to add VDS 17f7fe56-5f32-40ef-b098-611b84484e0b >2014-08-07 12:18:01,108 INFO [org.ovirt.engine.core.vdsbroker.AddVdsVDSCommand] (ajp--127.0.0.1-8702-2) [5d35448e] AddVds - VDS 17f7fe56-5f32-40ef-b098-611b84484e0b was added, will try to add it to the resource manager >2014-08-07 12:18:01,109 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (ajp--127.0.0.1-8702-2) [5d35448e] Entered VdsManager constructor >2014-08-07 12:18:01,130 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (ajp--127.0.0.1-8702-2) [5d35448e] Initialize vdsBroker (10.34.62.208,54,321) >2014-08-07 12:18:01,134 INFO [org.ovirt.engine.core.vdsbroker.ResourceManager] (ajp--127.0.0.1-8702-2) [5d35448e] VDS 17f7fe56-5f32-40ef-b098-611b84484e0b was added to the Resource Manager >2014-08-07 12:18:01,136 INFO [org.ovirt.engine.core.vdsbroker.AddVdsVDSCommand] (ajp--127.0.0.1-8702-2) [5d35448e] FINISH, AddVdsVDSCommand, log id: 4e3cd999 >2014-08-07 12:18:01,154 INFO [org.ovirt.engine.core.bll.AddVdsSpmIdCommand] (ajp--127.0.0.1-8702-2) [2f7b25df] Running command: AddVdsSpmIdCommand internal: true. Entities affected : ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS >2014-08-07 12:18:01,166 INFO [org.ovirt.engine.core.bll.AddVdsSpmIdCommand] (ajp--127.0.0.1-8702-2) [2f7b25df] Lock freed to object EngineLock [exclusiveLocks= key: 00000002-0002-0002-0002-0000000001d9 value: REGISTER_VDS >, sharedLocks= ] >2014-08-07 12:18:01,197 INFO [org.ovirt.engine.core.bll.network.host.PersistentSetupNetworksCommand] (org.ovirt.thread.pool-8-thread-40) [1511d738] Running command: PersistentSetupNetworksCommand internal: true. Entities affected : ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS >2014-08-07 12:18:01,208 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-2) [2f7b25df] Correlation ID: 5d35448e, Call Stack: null, Custom Event ID: -1, Message: Host rack configuration was updated by admin. >2014-08-07 12:18:01,247 INFO [org.ovirt.engine.core.bll.network.host.SetupNetworksCommand] (org.ovirt.thread.pool-8-thread-40) [1511d738] Running command: SetupNetworksCommand internal: true. Entities affected : ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS >2014-08-07 12:18:01,248 INFO [org.ovirt.engine.core.bll.network.host.SetupNetworksCommand] (org.ovirt.thread.pool-8-thread-40) [1511d738] No changes were detected in setup networks for host rack (ID: 17f7fe56-5f32-40ef-b098-611b84484e0b) >2014-08-07 12:18:04,136 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-65) [126267c5] Initializing Host: rack >2014-08-07 12:18:06,465 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-6) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-07 12:18:06,466 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-6) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-07 12:18:06,467 INFO [org.ovirt.engine.core.bll.UpdateVdsCommand] (ajp--127.0.0.1-8702-6) [6f7ae589] Running command: UpdateVdsCommand internal: false. Entities affected : ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS >2014-08-07 12:18:06,481 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-6) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-07 12:18:06,482 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-6) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-07 12:18:06,489 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) [6f7ae589] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host rack. >2014-08-07 12:18:06,496 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) [6f7ae589] Correlation ID: 6f7ae589, Call Stack: null, Custom Event ID: -1, Message: Host rack configuration was updated by admin. >2014-08-07 12:30:55,010 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-43) Principal admin::internal synchronized >2014-08-07 12:30:55,810 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-74) Attempting to update VMs/Templates Ovf. >2014-08-07 13:13:25,248 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (ajp--127.0.0.1-8702-11) [69a5df17] Lock Acquired to object EngineLock [exclusiveLocks= key: 17f7fe56-5f32-40ef-b098-611b84484e0b value: VDS >, sharedLocks= ] >2014-08-07 13:13:25,274 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (org.ovirt.thread.pool-8-thread-1) [69a5df17] Running command: ActivateVdsCommand internal: false. Entities affected : ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS >2014-08-07 13:13:25,275 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (org.ovirt.thread.pool-8-thread-1) [69a5df17] Before acquiring lock in order to prevent monitoring for host rack from data-center Default >2014-08-07 13:13:25,276 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (org.ovirt.thread.pool-8-thread-1) [69a5df17] Lock acquired, from now a monitoring of host will be skipped for host rack from data-center Default >2014-08-07 13:13:25,282 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-8-thread-1) [69a5df17] START, SetVdsStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 6079145b >2014-08-07 13:13:25,291 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-8-thread-1) [69a5df17] FINISH, SetVdsStatusVDSCommand, log id: 6079145b >2014-08-07 13:13:25,348 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (org.ovirt.thread.pool-8-thread-1) [69a5df17] Activate finished. Lock released. Monitoring can run now for host rack from data-center Default >2014-08-07 13:13:25,377 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-1) [69a5df17] Correlation ID: 69a5df17, Job ID: 0d45ef1f-8908-4172-b6d3-c2759d8fa2ec, Call Stack: null, Custom Event ID: -1, Message: Host rack was activated by admin. >2014-08-07 13:13:25,389 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (org.ovirt.thread.pool-8-thread-1) [69a5df17] Lock freed to object EngineLock [exclusiveLocks= key: 17f7fe56-5f32-40ef-b098-611b84484e0b value: VDS >, sharedLocks= ] >2014-08-07 13:13:28,292 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-16) START, GetHardwareInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]), log id: 42c08c8e >2014-08-07 13:13:28,355 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-16) FINISH, GetHardwareInfoVDSCommand, log id: 42c08c8e >2014-08-07 13:13:28,381 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-16) [11ab7a6c] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS >2014-08-07 13:13:28,424 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-16) [11ab7a6c] START, GetHardwareInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]), log id: 31aaebd6 >2014-08-07 13:13:28,469 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-16) [11ab7a6c] FINISH, GetHardwareInfoVDSCommand, log id: 31aaebd6 >2014-08-07 13:13:28,533 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-16) [3194387f] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-08-07 13:13:28,542 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-16) [4e5d9f1d] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-08-07 13:13:28,553 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-16) [4e5d9f1d] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, storageType = NFS, connectionList = [{ id: 05047c9a-152e-4ef2-b118-4a1753c1eae0, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35export, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 7443bab0 >2014-08-07 13:13:28,703 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-16) [4e5d9f1d] FINISH, ConnectStorageServerVDSCommand, return: {5083f370-af09-457b-9b73-3440e8b65a07=0, 05047c9a-152e-4ef2-b118-4a1753c1eae0=0}, log id: 7443bab0 >2014-08-07 13:13:28,706 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-16) [4e5d9f1d] Host rack storage connection was succeeded >2014-08-07 13:13:28,713 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-46) START, ConnectStoragePoolVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, masterVersion = 1), log id: 186a7137 >2014-08-07 13:13:52,837 INFO [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (ajp--127.0.0.1-8702-11) [54f96cf8] Lock Acquired to object EngineLock [exclusiveLocks= key: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 value: STORAGE >, sharedLocks= ] >2014-08-07 13:13:52,844 WARN [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (ajp--127.0.0.1-8702-11) [54f96cf8] CanDoAction of action ActivateStorageDomain failed. Reasons:VAR__TYPE__STORAGE__DOMAIN,VAR__ACTION__ACTIVATE,ACTION_TYPE_FAILED_NO_VDS_IN_POOL >2014-08-07 13:13:52,846 INFO [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (ajp--127.0.0.1-8702-11) [54f96cf8] Lock freed to object EngineLock [exclusiveLocks= key: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 value: STORAGE >, sharedLocks= ] >2014-08-07 13:14:29,174 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-46) FINISH, ConnectStoragePoolVDSCommand, log id: 186a7137 >2014-08-07 13:14:29,193 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-16) [4e5d9f1d] START, SetMOMPolicyParametersVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 33fb4bec >2014-08-07 13:14:29,345 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-16) [4e5d9f1d] FINISH, SetMOMPolicyParametersVDSCommand, log id: 33fb4bec >2014-08-07 13:14:29,358 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-16) [4e5d9f1d] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host rack was set to Up. >2014-08-07 13:14:29,364 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-16) [4e5d9f1d] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host rack. >2014-08-07 13:14:29,381 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-16) [53385366] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS >2014-08-07 13:14:29,486 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-53) hostFromVds::selectedVds - rack, spmStatus Free, storage pool Default >2014-08-07 13:14:29,492 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-53) starting spm on vds rack, storage pool Default, prevId -1, LVER -1 >2014-08-07 13:14:29,495 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-53) START, SpmStartVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, prevId=-1, prevLVER=-1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 5d9ba7e5 >2014-08-07 13:14:29,544 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-53) spmStart polling started: taskId = 60a9bf0d-9f4c-4468-b4c3-547edf64be16 >2014-08-07 13:14:30,551 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-53) spmStart polling ended: taskId = 60a9bf0d-9f4c-4468-b4c3-547edf64be16 task status = finished >2014-08-07 13:14:30,607 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-53) spmStart polling ended, spm status: SPM >2014-08-07 13:14:30,609 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-53) START, HSMClearTaskVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, taskId=60a9bf0d-9f4c-4468-b4c3-547edf64be16), log id: 7c1c6b59 >2014-08-07 13:14:30,654 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-53) FINISH, HSMClearTaskVDSCommand, log id: 7c1c6b59 >2014-08-07 13:14:30,654 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-53) FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@7a8b3539, log id: 5d9ba7e5 >2014-08-07 13:14:30,658 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-53) Initialize Irs proxy from vds: 10.34.62.208 >2014-08-07 13:14:30,665 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-53) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208). >2014-08-07 13:14:30,678 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-16) START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 57693b23 >2014-08-07 13:14:30,706 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-16) -- executeIrsBrokerCommand: Attempting on storage pool 00000002-0002-0002-0002-0000000001d9 >2014-08-07 13:14:30,708 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-16) START, HSMGetAllTasksInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 1ba620ff >2014-08-07 13:14:30,713 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-16) FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 1ba620ff >2014-08-07 13:14:30,714 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-16) FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 57693b23 >2014-08-07 13:14:30,715 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (org.ovirt.thread.pool-8-thread-16) Discovered no tasks on Storage Pool Default >2014-08-07 13:14:54,761 INFO [org.ovirt.engine.core.bll.storage.AttachStorageDomainToPoolCommand] (org.ovirt.thread.pool-8-thread-17) [76d232c7] Running command: AttachStorageDomainToPoolCommand internal: false. Entities affected : ID: 63fe3516-4170-402a-9dc8-6d3eab1c7933 Type: Storage >2014-08-07 13:14:54,780 INFO [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-8-thread-41) [656265fc] Running command: ConnectStorageToVdsCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2014-08-07 13:14:54,784 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-41) [656265fc] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: 7fe036a2-2dfc-4565-b936-69a678cf0613, connection: pb-rh35.rhev.lab.eng.brq.redhat.com:/var/lib/exports/iso, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 7e1e952a >2014-08-07 13:17:04,848 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-41) [656265fc] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host rack to the Storage Domains ISO_DOMAIN. >2014-08-07 13:17:04,849 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-41) [656265fc] FINISH, ConnectStorageServerVDSCommand, return: {7fe036a2-2dfc-4565-b936-69a678cf0613=477}, log id: 7e1e952a >2014-08-07 13:17:04,852 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-41) [656265fc] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: The error message for connection pb-rh35.rhev.lab.eng.brq.redhat.com:/var/lib/exports/iso returned by VDSM was: Problem while trying to mount target >2014-08-07 13:17:04,854 ERROR [org.ovirt.engine.core.bll.storage.NFSStorageHelper] (org.ovirt.thread.pool-8-thread-41) [656265fc] The connection with details pb-rh35.rhev.lab.eng.brq.redhat.com:/var/lib/exports/iso failed because of error code 477 and error message is: problem while trying to mount target >2014-08-07 13:17:04,859 ERROR [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-8-thread-41) [656265fc] Transaction rolled-back for command: org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand. >2014-08-07 13:17:04,861 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.AttachStorageDomainVDSCommand] (org.ovirt.thread.pool-8-thread-17) [76d232c7] START, AttachStorageDomainVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, storageDomainId = 63fe3516-4170-402a-9dc8-6d3eab1c7933), log id: 2b849a32 >2014-08-07 13:19:05,282 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.AttachStorageDomainVDSCommand] (org.ovirt.thread.pool-8-thread-17) [76d232c7] Failed in AttachStorageDomainVDS method >2014-08-07 13:19:05,342 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (org.ovirt.thread.pool-8-thread-17) [76d232c7] IrsBroker::Failed::AttachStorageDomainVDS due to: IRSErrorException: IRSGenericException: IRSErrorException: Failed to AttachStorageDomainVDS, error = Storage domain does not exist: ('63fe3516-4170-402a-9dc8-6d3eab1c7933',), code = 358 >2014-08-07 13:19:05,346 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.AttachStorageDomainVDSCommand] (org.ovirt.thread.pool-8-thread-17) [76d232c7] FINISH, AttachStorageDomainVDSCommand, log id: 2b849a32 >2014-08-07 13:19:05,348 ERROR [org.ovirt.engine.core.bll.storage.AttachStorageDomainToPoolCommand] (org.ovirt.thread.pool-8-thread-17) [76d232c7] Command org.ovirt.engine.core.bll.storage.AttachStorageDomainToPoolCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.irsbroker.IRSErrorException: IRSGenericException: IRSErrorException: Failed to AttachStorageDomainVDS, error = Storage domain does not exist: ('63fe3516-4170-402a-9dc8-6d3eab1c7933',), code = 358 (Failed with error StorageDomainDoesNotExist and code 358) >2014-08-07 13:19:05,373 INFO [org.ovirt.engine.core.bll.storage.AttachStorageDomainToPoolCommand] (org.ovirt.thread.pool-8-thread-17) [76d232c7] Command [id=84a2ceb4-886a-4665-a8fb-56449a88a69a]: Compensating NEW_ENTITY_ID of org.ovirt.engine.core.common.businessentities.StoragePoolIsoMap; snapshot: storagePoolId = 00000002-0002-0002-0002-0000000001d9, storageId = 63fe3516-4170-402a-9dc8-6d3eab1c7933. >2014-08-07 13:19:05,393 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-17) [76d232c7] Correlation ID: 76d232c7, Job ID: 09bed423-fc5d-4cae-a270-aeceba1112d1, Call Stack: null, Custom Event ID: -1, Message: Failed to attach Storage Domain ISO_DOMAIN to Data Center Default. (User: admin) >2014-08-07 13:30:55,017 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-99) Principal admin::internal synchronized >2014-08-07 13:30:55,814 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-92) Attempting to update VMs/Templates Ovf. >2014-08-07 13:51:49,118 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-9) Running command: LoginUserCommand internal: false. >2014-08-07 13:51:49,126 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-9) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-08-07 13:53:38,697 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-10) Running command: LoginUserCommand internal: false. >2014-08-07 13:53:38,702 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-10) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-08-07 13:54:15,483 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-6) Running command: LoginUserCommand internal: false. >2014-08-07 13:54:15,487 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-08-07 13:57:04,806 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-11) Running command: LoginUserCommand internal: false. >2014-08-07 13:57:04,815 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-11) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-08-07 13:57:19,925 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-2) Running command: LoginUserCommand internal: false. >2014-08-07 13:57:19,930 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-2) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-08-07 13:58:06,010 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-11) Running command: LoginUserCommand internal: false. >2014-08-07 13:58:06,013 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-11) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-08-07 13:58:20,611 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-2) Running command: LoginUserCommand internal: false. >2014-08-07 13:58:20,615 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-2) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-08-07 13:58:21,820 INFO [org.ovirt.engine.core.bll.AddVmFromSnapshotCommand] (ajp--127.0.0.1-8702-6) [1209e408] Lock Acquired to object EngineLock [exclusiveLocks= key: newVM value: VM_NAME >key: 78722009-2727-4715-91bf-dbbb33ce111e value: VM >, sharedLocks= ] >2014-08-07 13:58:21,900 INFO [org.ovirt.engine.core.bll.AddVmFromSnapshotCommand] (ajp--127.0.0.1-8702-6) [1209e408] Running command: AddVmFromSnapshotCommand internal: false. Entities affected : ID: 00000001-0001-0001-0001-0000000002c2 Type: VdsGroups, ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage, ID: 78722009-2727-4715-91bf-dbbb33ce111e Type: VM >2014-08-07 13:58:21,962 INFO [org.ovirt.engine.core.bll.AddVmFromSnapshotCommand] (ajp--127.0.0.1-8702-6) [1209e408] Locking VM(id = 10a64f7a-b891-46c7-be45-36a8d2b0a754) with compensation. >2014-08-07 13:58:21,965 INFO [org.ovirt.engine.core.vdsbroker.SetVmStatusVDSCommand] (ajp--127.0.0.1-8702-6) [1209e408] START, SetVmStatusVDSCommand( vmId = 10a64f7a-b891-46c7-be45-36a8d2b0a754, status = ImageLocked, exit status = Normal), log id: 2fa884d3 >2014-08-07 13:58:21,974 INFO [org.ovirt.engine.core.vdsbroker.SetVmStatusVDSCommand] (ajp--127.0.0.1-8702-6) [1209e408] FINISH, SetVmStatusVDSCommand, log id: 2fa884d3 >2014-08-07 13:58:21,978 INFO [org.ovirt.engine.core.bll.AddVmFromSnapshotCommand] (ajp--127.0.0.1-8702-6) [1209e408] Lock freed to object EngineLock [exclusiveLocks= key: newVM value: VM_NAME >key: 78722009-2727-4715-91bf-dbbb33ce111e value: VM >, sharedLocks= ] >2014-08-07 13:58:22,001 INFO [org.ovirt.engine.core.bll.CopyImageGroupCommand] (ajp--127.0.0.1-8702-6) [41881ba1] Running command: CopyImageGroupCommand internal: true. Entities affected : ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage >2014-08-07 13:58:22,285 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.CopyImageVDSCommand] (ajp--127.0.0.1-8702-6) [41881ba1] START, CopyImageVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, storageDomainId = 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752, imageGroupId = 6cae69b6-2d4f-4e61-993c-dce113dd19e5, imageId = b05d9fa4-2cf0-414e-b031-f89fcdf2cf0b, dstImageGroupId = 348a2b24-96f9-4d7c-93bd-771f95224b0b, vmId = 10a64f7a-b891-46c7-be45-36a8d2b0a754, dstImageId = d0fd3f56-e0a1-495c-b41b-dfa272854235, imageDescription = , dstStorageDomainId = 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752, copyVolumeType = LeafVol, volumeFormat = RAW, preallocate = Sparse, postZero = false, force = false), log id: 337b3ddf >2014-08-07 13:58:22,291 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.CopyImageVDSCommand] (ajp--127.0.0.1-8702-6) [41881ba1] -- executeIrsBrokerCommand: calling 'copyImage' with two new parameters: description and UUID >2014-08-07 13:58:22,292 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.CopyImageVDSCommand] (ajp--127.0.0.1-8702-6) [41881ba1] -- copyImage parameters: > sdUUID=71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 > spUUID=00000002-0002-0002-0002-0000000001d9 > vmGUID=10a64f7a-b891-46c7-be45-36a8d2b0a754 > srcImageGUID=6cae69b6-2d4f-4e61-993c-dce113dd19e5 > srcVolUUID=b05d9fa4-2cf0-414e-b031-f89fcdf2cf0b > dstImageGUID=348a2b24-96f9-4d7c-93bd-771f95224b0b > dstVolUUID=d0fd3f56-e0a1-495c-b41b-dfa272854235 > descr= > dstSdUUID=71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 >2014-08-07 13:58:22,744 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.CopyImageVDSCommand] (ajp--127.0.0.1-8702-6) [41881ba1] FINISH, CopyImageVDSCommand, return: 00000000-0000-0000-0000-000000000000, log id: 337b3ddf >2014-08-07 13:58:22,813 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (ajp--127.0.0.1-8702-6) [41881ba1] CommandAsyncTask::Adding CommandMultiAsyncTasks object for command f99f4bac-e57f-4749-b26e-751c2a3b26e7 >2014-08-07 13:58:22,814 INFO [org.ovirt.engine.core.bll.CommandMultiAsyncTasks] (ajp--127.0.0.1-8702-6) [41881ba1] CommandMultiAsyncTasks::AttachTask: Attaching task 7b1882c9-f22a-4ede-b8fc-ad390dfe51bb to command f99f4bac-e57f-4749-b26e-751c2a3b26e7. >2014-08-07 13:58:22,850 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (ajp--127.0.0.1-8702-6) [41881ba1] Adding task 7b1882c9-f22a-4ede-b8fc-ad390dfe51bb (Parent Command AddVmFromSnapshot, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters), polling hasn't started yet.. >2014-08-07 13:58:22,982 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) [41881ba1] Correlation ID: 1209e408, Job ID: 44e11b7d-d743-4d68-95f6-d691deaf5aa1, Call Stack: null, Custom Event ID: -1, Message: VM newVM creation was initiated by admin. >2014-08-07 13:58:22,983 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (ajp--127.0.0.1-8702-6) [41881ba1] BaseAsyncTask::startPollingTask: Starting to poll task 7b1882c9-f22a-4ede-b8fc-ad390dfe51bb. >2014-08-07 13:58:28,125 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (DefaultQuartzScheduler_Worker-57) [699d5b35] Polling and updating Async Tasks: 1 tasks, 1 tasks to poll now >2014-08-07 13:58:28,133 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (DefaultQuartzScheduler_Worker-57) [699d5b35] SPMAsyncTask::PollTask: Polling task 7b1882c9-f22a-4ede-b8fc-ad390dfe51bb (Parent Command AddVmFromSnapshot, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) returned status running. >2014-08-07 13:58:28,135 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (DefaultQuartzScheduler_Worker-57) [699d5b35] Finished polling Tasks, will poll again in 10 seconds. >2014-08-07 14:00:21,368 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-57) [699d5b35] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host rack has network interface which exceeded the defined threshold [95%] (em1: transmit rate[0%], receive rate [98%]) >2014-08-07 14:06:04,423 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (DefaultQuartzScheduler_Worker-73) SPMAsyncTask::PollTask: Polling task 7b1882c9-f22a-4ede-b8fc-ad390dfe51bb (Parent Command AddVmFromSnapshot, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) returned status finished, result 'success'. >2014-08-07 14:06:04,435 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (DefaultQuartzScheduler_Worker-73) BaseAsyncTask::OnTaskEndSuccess: Task 7b1882c9-f22a-4ede-b8fc-ad390dfe51bb (Parent Command AddVmFromSnapshot, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) ended successfully. >2014-08-07 14:06:04,437 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (DefaultQuartzScheduler_Worker-73) CommandAsyncTask::EndActionIfNecessary: All tasks of command f99f4bac-e57f-4749-b26e-751c2a3b26e7 has ended -> executing endAction >2014-08-07 14:06:04,439 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (DefaultQuartzScheduler_Worker-73) CommandAsyncTask::endAction: Ending action for 1 tasks (command ID: f99f4bac-e57f-4749-b26e-751c2a3b26e7): calling endAction . >2014-08-07 14:06:04,442 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-8-thread-37) CommandAsyncTask::EndCommandAction [within thread] context: Attempting to endAction AddVmFromSnapshot, executionIndex: 0 >2014-08-07 14:06:04,526 INFO [org.ovirt.engine.core.bll.AddVmFromSnapshotCommand] (org.ovirt.thread.pool-8-thread-37) [1209e408] Ending command successfully: org.ovirt.engine.core.bll.AddVmFromSnapshotCommand >2014-08-07 14:06:04,636 INFO [org.ovirt.engine.core.bll.CopyImageGroupCommand] (org.ovirt.thread.pool-8-thread-37) [41881ba1] Ending command successfully: org.ovirt.engine.core.bll.CopyImageGroupCommand >2014-08-07 14:06:04,644 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetImageInfoVDSCommand] (org.ovirt.thread.pool-8-thread-37) [41881ba1] START, GetImageInfoVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, storageDomainId = 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752, imageGroupId = 348a2b24-96f9-4d7c-93bd-771f95224b0b, imageId = d0fd3f56-e0a1-495c-b41b-dfa272854235), log id: 5019143b >2014-08-07 14:06:04,677 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetImageInfoVDSCommand] (org.ovirt.thread.pool-8-thread-37) [41881ba1] FINISH, GetImageInfoVDSCommand, return: org.ovirt.engine.core.common.businessentities.DiskImage@de298e57, log id: 5019143b >2014-08-07 14:06:04,698 INFO [org.ovirt.engine.core.vdsbroker.SetVmStatusVDSCommand] (org.ovirt.thread.pool-8-thread-37) START, SetVmStatusVDSCommand( vmId = 10a64f7a-b891-46c7-be45-36a8d2b0a754, status = Down, exit status = Normal), log id: 3f71eefb >2014-08-07 14:06:04,706 INFO [org.ovirt.engine.core.vdsbroker.SetVmStatusVDSCommand] (org.ovirt.thread.pool-8-thread-37) FINISH, SetVmStatusVDSCommand, log id: 3f71eefb >2014-08-07 14:06:04,756 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-37) Correlation ID: 1209e408, Call Stack: null, Custom Event ID: -1, Message: VM newVM creation has been completed. >2014-08-07 14:06:04,757 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-8-thread-37) CommandAsyncTask::HandleEndActionResult [within thread]: endAction for action type AddVmFromSnapshot completed, handling the result. >2014-08-07 14:06:04,758 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-8-thread-37) CommandAsyncTask::HandleEndActionResult [within thread]: endAction for action type AddVmFromSnapshot succeeded, clearing tasks. >2014-08-07 14:06:04,764 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (org.ovirt.thread.pool-8-thread-37) SPMAsyncTask::ClearAsyncTask: Attempting to clear task 7b1882c9-f22a-4ede-b8fc-ad390dfe51bb >2014-08-07 14:06:04,766 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-37) START, SPMClearTaskVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, taskId = 7b1882c9-f22a-4ede-b8fc-ad390dfe51bb), log id: 1e39fb86 >2014-08-07 14:06:04,769 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-37) START, HSMClearTaskVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, taskId=7b1882c9-f22a-4ede-b8fc-ad390dfe51bb), log id: 70b029ea >2014-08-07 14:06:04,820 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-37) FINISH, HSMClearTaskVDSCommand, log id: 70b029ea >2014-08-07 14:06:04,821 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-37) FINISH, SPMClearTaskVDSCommand, log id: 1e39fb86 >2014-08-07 14:06:04,838 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (org.ovirt.thread.pool-8-thread-37) BaseAsyncTask::RemoveTaskFromDB: Removed task 7b1882c9-f22a-4ede-b8fc-ad390dfe51bb from DataBase >2014-08-07 14:06:04,841 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-8-thread-37) CommandAsyncTask::HandleEndActionResult [within thread]: Removing CommandMultiAsyncTasks object for entity f99f4bac-e57f-4749-b26e-751c2a3b26e7 >2014-08-07 14:07:25,906 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (DefaultQuartzScheduler_Worker-10) Setting new tasks map. The map contains now 0 tasks >2014-08-07 14:07:25,909 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (DefaultQuartzScheduler_Worker-10) Cleared all tasks of pool 00000002-0002-0002-0002-0000000001d9. >2014-08-07 14:09:27,243 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-41) domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 in problem. vds: rack >2014-08-07 14:09:27,246 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-41) domain 44559ec0-6bef-491c-bbd6-e67278f37987:export in problem. vds: rack >2014-08-07 14:10:55,898 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-3) Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2014-08-07 14:10:55,903 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-3) [301effc7] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-08-07 14:10:55,906 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-3) [301effc7] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Active to Unknown, reason : null >2014-08-07 14:10:55,909 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-3) [301effc7] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Active to Unknown, reason : null >2014-08-07 14:10:55,924 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-3) [301effc7] Correlation ID: 301effc7, Call Stack: null, Custom Event ID: -1, Message: Invalid status on Data Center Default. Setting Data Center status to Non Responsive (On host rack, Error: Cannot access master storage domain). >2014-08-07 14:12:55,951 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-3) [301effc7] Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2014-08-07 14:12:55,953 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-3) [301effc7] hostFromVds::selectedVds - rack, spmStatus returned null! >2014-08-07 14:12:55,954 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-3) [301effc7] IrsBroker::Failed::GetStoragePoolInfoVDS >2014-08-07 14:12:55,955 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-3) [301effc7] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2014-08-07 14:12:55,991 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-4) [d1e91b4] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage >2014-08-07 14:12:56,010 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-4) [d1e91b4] START, ResetIrsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, ignoreStopFailed = true), log id: 4a63088b >2014-08-07 14:12:56,013 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-4) [d1e91b4] START, SpmStopVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9), log id: 30b01c9 >2014-08-07 14:12:56,019 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-4) [d1e91b4] SpmStopVDSCommand::Stopping SPM on vds rack, pool id 00000002-0002-0002-0002-0000000001d9 >2014-08-07 14:14:55,921 WARN [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-4) [d1e91b4] could not stop spm of pool 00000002-0002-0002-0002-0000000001d9 on vds 17f7fe56-5f32-40ef-b098-611b84484e0b - reason: org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcRunTimeException: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException >2014-08-07 14:14:55,931 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ListVDSCommand] (DefaultQuartzScheduler_Worker-87) Command ListVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-08-07 14:14:55,933 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-83) Command SpmStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9) execution failed. Exception: VDSNetworkException: org.apache.commons.httpclient.NoHttpResponseException: The server 10.34.62.208 failed to respond >2014-08-07 14:14:55,937 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-83) hostFromVds::selectedVds - rack, spmStatus returned null! >2014-08-07 14:14:55,932 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-4) [d1e91b4] FINISH, SpmStopVDSCommand, log id: 30b01c9 >2014-08-07 14:14:55,934 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-87) Host rack is not responding. It will stay in Connecting state for a grace period of $80 seconds and after that an attempt to fence the host will be issued. >2014-08-07 14:14:55,942 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-87) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:407) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.fetchRunningVms(VdsUpdateRunTimeInfo.java:971) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVmStats(VdsUpdateRunTimeInfo.java:920) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:638) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor17.invoke(Unknown Source) [:1.7.0_55] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_55] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_55] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_55] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_55] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_55] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_55] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_55] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy83.list(Unknown Source) > at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source) [:1.7.0_55] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_55] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_55] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_55] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_55] > >2014-08-07 14:14:55,952 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-4) [d1e91b4] FINISH, ResetIrsVDSCommand, log id: 4a63088b >2014-08-07 14:14:55,990 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-87) Failed to refresh VDS , vds = 17f7fe56-5f32-40ef-b098-611b84484e0b : rack, VDS Network Error, continuing. >java.net.ConnectException: Connection refused >2014-08-07 14:14:55,993 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-4) [d1e91b4] Correlation ID: d1e91b4, Job ID: 5525121d-f1d6-4b7b-b002-594242ccff24, Call Stack: null, Custom Event ID: -1, Message: Failed to Reconstruct Master Domain for Data Center Default. >2014-08-07 14:14:56,004 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-49) [301effc7] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue >2014-08-07 14:14:56,006 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-2) starting processDomainRecovery for domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 >2014-08-07 14:14:56,071 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-2) Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752:35 was reported by all hosts in status UP as problematic. Not moving the domain to NonOperational because it is being reconstructed now. >2014-08-07 14:14:56,089 WARN [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-2) [ce514ae] CanDoAction of action ReconstructMasterDomain failed. Reasons:VAR__ACTION__RECONSTRUCT_MASTER,VAR__TYPE__STORAGE__DOMAIN,ACTION_TYPE_FAILED_NO_VDS_IN_POOL >2014-08-07 14:14:56,096 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-2) [ce514ae] Removing vds [] from the domain in maintenance cache >2014-08-07 14:14:56,097 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-2) [ce514ae] Removing host(s) [] from hosts unseen domain report cache >2014-08-07 14:14:56,098 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-49) [301effc7] Finished reconstruct for pool 00000002-0002-0002-0002-0000000001d9. Clearing event queue >2014-08-07 14:14:56,099 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-36) starting processDomainRecovery for domain 44559ec0-6bef-491c-bbd6-e67278f37987:export >2014-08-07 14:14:56,104 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-36) Domain 44559ec0-6bef-491c-bbd6-e67278f37987:export was reported by all hosts in status UP as problematic. Moving the domain to NonOperational. >2014-08-07 14:14:56,121 INFO [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-36) [ecf806c] Lock Acquired to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE >, sharedLocks= ] >2014-08-07 14:14:56,129 INFO [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-36) [ecf806c] Running command: DeactivateStorageDomainCommand internal: true. Entities affected : ID: 44559ec0-6bef-491c-bbd6-e67278f37987 Type: Storage >2014-08-07 14:14:56,135 INFO [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-36) [ecf806c] Lock freed to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE >, sharedLocks= ] >2014-08-07 14:14:56,145 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-36) [ecf806c] Correlation ID: ecf806c, Job ID: 457b2c2b-2326-4352-980c-9e4808604c03, Call Stack: null, Custom Event ID: -1, Message: Storage Domain export (Data Center Default) was deactivated by system because it's not visible by any of the hosts. >2014-08-07 14:14:56,151 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-36) [ecf806c] Removing vds [] from the domain in maintenance cache >2014-08-07 14:14:56,152 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-36) [ecf806c] Removing host(s) [] from hosts unseen domain report cache >2014-08-07 14:14:59,606 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-9) START, GetHardwareInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]), log id: 4a33b6d9 >2014-08-07 14:14:59,659 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-9) FINISH, GetHardwareInfoVDSCommand, log id: 4a33b6d9 >2014-08-07 14:14:59,688 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-9) [3f132663] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS >2014-08-07 14:14:59,740 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-9) [4151a43a] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-08-07 14:14:59,748 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-9) [598afe80] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-08-07 14:14:59,761 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-9) [598afe80] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, storageType = NFS, connectionList = [{ id: 05047c9a-152e-4ef2-b118-4a1753c1eae0, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35export, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 4e4d6360 >2014-08-07 14:14:59,827 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-9) [598afe80] FINISH, ConnectStorageServerVDSCommand, return: {5083f370-af09-457b-9b73-3440e8b65a07=0, 05047c9a-152e-4ef2-b118-4a1753c1eae0=0}, log id: 4e4d6360 >2014-08-07 14:14:59,828 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-9) [598afe80] Host rack storage connection was succeeded >2014-08-07 14:14:59,836 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-29) START, ConnectStoragePoolVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, masterVersion = 1), log id: 352a3b66 >2014-08-07 14:15:00,006 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-90) Autorecovering 2 storage domains >2014-08-07 14:15:00,006 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-90) Autorecovering storage domains id: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 >2014-08-07 14:15:00,009 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-90) [39f70177] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage >2014-08-07 14:15:00,010 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-90) [39f70177] ConnectDomainToStorage. Before Connect all hosts to pool. Time:8/7/14 2:15 PM >2014-08-07 14:15:00,016 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-90) [39f70177] ConnectDomainToStorage. After Connect all hosts to pool. Time:8/7/14 2:15 PM >2014-08-07 14:15:00,018 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-90) [39f70177] Autorecovering storage domains id: 44559ec0-6bef-491c-bbd6-e67278f37987 >2014-08-07 14:15:00,021 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-90) [f94de28] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 44559ec0-6bef-491c-bbd6-e67278f37987 Type: Storage >2014-08-07 14:15:00,022 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-90) [f94de28] ConnectDomainToStorage. Before Connect all hosts to pool. Time:8/7/14 2:15 PM >2014-08-07 14:15:00,032 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-90) [f94de28] ConnectDomainToStorage. After Connect all hosts to pool. Time:8/7/14 2:15 PM >2014-08-07 14:15:05,943 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-61) Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Inactive to Unknown, reason : no reporting hosts >2014-08-07 14:15:05,947 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-61) Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Inactive to Unknown, reason : no reporting hosts >2014-08-07 14:16:00,268 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-29) FINISH, ConnectStoragePoolVDSCommand, log id: 352a3b66 >2014-08-07 14:16:00,282 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-9) [598afe80] START, SetMOMPolicyParametersVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 4d5be635 >2014-08-07 14:16:00,375 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-9) [598afe80] FINISH, SetMOMPolicyParametersVDSCommand, log id: 4d5be635 >2014-08-07 14:16:00,383 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-9) [598afe80] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host rack was set to Up. >2014-08-07 14:16:00,389 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-9) [598afe80] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host rack. >2014-08-07 14:16:00,407 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-9) [2512845a] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS >2014-08-07 14:16:06,094 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-96) hostFromVds::selectedVds - rack, spmStatus Free, storage pool Default >2014-08-07 14:16:06,100 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-96) starting spm on vds rack, storage pool Default, prevId -1, LVER -1 >2014-08-07 14:16:06,102 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-96) START, SpmStartVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, prevId=-1, prevLVER=-1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 2de7461b >2014-08-07 14:16:06,150 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-96) spmStart polling started: taskId = 08a4a8e1-a14e-4900-a42a-117a0fbfab0a >2014-08-07 14:16:07,157 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-96) spmStart polling ended: taskId = 08a4a8e1-a14e-4900-a42a-117a0fbfab0a task status = finished >2014-08-07 14:16:07,216 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-96) spmStart polling ended, spm status: SPM >2014-08-07 14:16:07,219 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-96) START, HSMClearTaskVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, taskId=08a4a8e1-a14e-4900-a42a-117a0fbfab0a), log id: 22b925d0 >2014-08-07 14:16:07,264 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-96) FINISH, HSMClearTaskVDSCommand, log id: 22b925d0 >2014-08-07 14:16:07,265 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-96) FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@4c22f65e, log id: 2de7461b >2014-08-07 14:16:07,273 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-96) Initialize Irs proxy from vds: 10.34.62.208 >2014-08-07 14:16:07,279 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-96) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208). >2014-08-07 14:16:07,308 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-39) START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 3a9705a8 >2014-08-07 14:16:07,330 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-39) -- executeIrsBrokerCommand: Attempting on storage pool 00000002-0002-0002-0002-0000000001d9 >2014-08-07 14:16:07,334 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-39) START, HSMGetAllTasksInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 31455495 >2014-08-07 14:16:07,341 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-39) FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 31455495 >2014-08-07 14:16:07,343 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-39) FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 3a9705a8 >2014-08-07 14:16:07,344 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (org.ovirt.thread.pool-8-thread-39) Discovered no tasks on Storage Pool Default >2014-08-07 14:27:11,031 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Loaded file "/usr/share/ovirt-engine/services/ovirt-engine/ovirt-engine.conf". >2014-08-07 14:27:11,031 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) The file "/etc/ovirt-engine/engine.conf" doesn't exist or isn't readable. Will return an empty set of properties. >2014-08-07 14:27:11,032 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Loaded file "/etc/ovirt-engine/engine.conf.d/10-setup-database.conf". >2014-08-07 14:27:11,033 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Loaded file "/etc/ovirt-engine/engine.conf.d/10-setup-jboss.conf". >2014-08-07 14:27:11,034 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Loaded file "/etc/ovirt-engine/engine.conf.d/10-setup-pki.conf". >2014-08-07 14:27:11,034 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Loaded file "/etc/ovirt-engine/engine.conf.d/10-setup-protocols.conf". >2014-08-07 14:27:11,035 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_AJP_ENABLED" is "true". >2014-08-07 14:27:11,036 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_AJP_PORT" is "8702". >2014-08-07 14:27:11,036 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_APPS" is "engine.ear restapi.war legacy_restapi.war". >2014-08-07 14:27:11,037 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_CACHE" is "/var/cache/ovirt-engine". >2014-08-07 14:27:11,040 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_DB_CHECK_INTERVAL" is "1000". >2014-08-07 14:27:11,040 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_DB_CONNECTION_TIMEOUT" is "300000". >2014-08-07 14:27:11,041 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_DB_DATABASE" is "engine". >2014-08-07 14:27:11,041 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_DB_DRIVER" is "org.postgresql.Driver". >2014-08-07 14:27:11,042 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_DB_HOST" is "localhost". >2014-08-07 14:27:11,042 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_DB_MAX_CONNECTIONS" is "100". >2014-08-07 14:27:11,043 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_DB_MIN_CONNECTIONS" is "1". >2014-08-07 14:27:11,043 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_DB_PASSWORD" is "***". >2014-08-07 14:27:11,044 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_DB_PORT" is "5432". >2014-08-07 14:27:11,044 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_DB_SECURED" is "False". >2014-08-07 14:27:11,044 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_DB_SECURED_VALIDATION" is "False". >2014-08-07 14:27:11,045 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_DB_URL" is "jdbc:postgresql://localhost:5432/engine?sslfactory=org.postgresql.ssl.NonValidatingFactory". >2014-08-07 14:27:11,046 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_DB_USER" is "engine". >2014-08-07 14:27:11,046 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_DEBUG_ADDRESS" is "". >2014-08-07 14:27:11,046 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_DOC" is "/usr/share/doc/ovirt-engine". >2014-08-07 14:27:11,047 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_ETC" is "/etc/ovirt-engine". >2014-08-07 14:27:11,048 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_EXTENSION_PATH" is "/usr/share/ovirt-engine/extensions.d:/etc/ovirt-engine/extensions.d". >2014-08-07 14:27:11,049 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_FQDN" is "pb-rh35.rhev.lab.eng.brq.redhat.com". >2014-08-07 14:27:11,049 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_GROUP" is "ovirt". >2014-08-07 14:27:11,050 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_HEAP_MAX" is "1g". >2014-08-07 14:27:11,050 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_HEAP_MIN" is "1g". >2014-08-07 14:27:11,050 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_HTTPS_ENABLED" is "false". >2014-08-07 14:27:11,051 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_HTTPS_PORT" is "None". >2014-08-07 14:27:11,051 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_HTTPS_PROTOCOLS" is "SSLv3,TLSv1,TLSv1.1,TLSv1.2". >2014-08-07 14:27:11,052 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_HTTP_ENABLED" is "false". >2014-08-07 14:27:11,052 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_HTTP_PORT" is "None". >2014-08-07 14:27:11,053 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_JAVA_MODULEPATH" is "/usr/share/ovirt-engine/modules". >2014-08-07 14:27:11,053 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_JVM_ARGS" is " -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath="/var/log/ovirt-engine/dump"". >2014-08-07 14:27:11,054 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_LOG" is "/var/log/ovirt-engine". >2014-08-07 14:27:11,054 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_LOG_TO_CONSOLE" is "false". >2014-08-07 14:27:11,055 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_MANUAL" is "/usr/share/ovirt-engine/manual". >2014-08-07 14:27:11,055 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_PERM_MAX" is "256m". >2014-08-07 14:27:11,056 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_PERM_MIN" is "256m". >2014-08-07 14:27:11,056 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_PKI" is "/etc/pki/ovirt-engine". >2014-08-07 14:27:11,057 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_PKI_CA" is "/etc/pki/ovirt-engine/ca.pem". >2014-08-07 14:27:11,057 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_PKI_ENGINE_CERT" is "/etc/pki/ovirt-engine/certs/engine.cer". >2014-08-07 14:27:11,058 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_PKI_ENGINE_STORE" is "/etc/pki/ovirt-engine/keys/engine.p12". >2014-08-07 14:27:11,058 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_PKI_ENGINE_STORE_ALIAS" is "1". >2014-08-07 14:27:11,059 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_PKI_ENGINE_STORE_PASSWORD" is "***". >2014-08-07 14:27:11,059 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_PKI_TRUST_STORE" is "/etc/pki/ovirt-engine/.truststore". >2014-08-07 14:27:11,060 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_PKI_TRUST_STORE_PASSWORD" is "***". >2014-08-07 14:27:11,060 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_PROPERTIES" is " jsse.enableSNIExtension=false "java.security.krb5.conf=/etc/ovirt-engine/krb5.conf"". >2014-08-07 14:27:11,061 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_PROXY_ENABLED" is "true". >2014-08-07 14:27:11,061 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_PROXY_HTTPS_PORT" is "443". >2014-08-07 14:27:11,062 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_PROXY_HTTP_PORT" is "80". >2014-08-07 14:27:11,062 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_REPORTS_UI" is "/var/lib/ovirt-engine/reports.xml". >2014-08-07 14:27:11,063 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_STOP_INTERVAL" is "1". >2014-08-07 14:27:11,063 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_STOP_TIME" is "10". >2014-08-07 14:27:11,064 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_TMP" is "/var/tmp/ovirt-engine". >2014-08-07 14:27:11,064 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_UP_MARK" is "/var/lib/ovirt-engine/engine.up". >2014-08-07 14:27:11,065 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_URI" is "/ovirt-engine". >2014-08-07 14:27:11,065 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_USER" is "ovirt". >2014-08-07 14:27:11,065 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_USR" is "/usr/share/ovirt-engine". >2014-08-07 14:27:11,066 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_VAR" is "/var/lib/ovirt-engine". >2014-08-07 14:27:11,066 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_VERBOSE_GC" is "false". >2014-08-07 14:27:11,067 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "JBOSS_HOME" is "/usr/share/ovirt-engine-jboss-as". >2014-08-07 14:27:11,067 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "JBOSS_RUNTIME" is "/var/lib/ovirt-engine/jboss_runtime". >2014-08-07 14:27:11,068 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "SENSITIVE_KEYS" is ",ENGINE_DB_PASSWORD,ENGINE_PKI_TRUST_STORE_PASSWORD,ENGINE_PKI_ENGINE_STORE_PASSWORD". >2014-08-07 14:27:11,996 INFO [org.ovirt.engine.core.bll.Backend] (MSC service thread 1-1) Start initializing Backend >2014-08-07 14:27:12,799 INFO [org.ovirt.engine.core.utils.osinfo.OsInfoPreferencesLoader] (MSC service thread 1-1) Loaded file /etc/ovirt-engine/osinfo.conf.d/00-defaults.properties >2014-08-07 14:27:12,936 INFO [org.ovirt.engine.core.bll.Backend] (MSC service thread 1-1) Running ovirt-engine 3.5.0-0.0.master.20140804172041.git23b558e.el6 >2014-08-07 14:27:12,937 INFO [org.ovirt.engine.core.bll.CpuFlagsManagerHandler] (MSC service thread 1-1) Start initializing dictionaries >2014-08-07 14:27:12,941 INFO [org.ovirt.engine.core.bll.CpuFlagsManagerHandler] (MSC service thread 1-1) Finished initializing dictionaries >2014-08-07 14:27:12,941 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (MSC service thread 1-1) Start initializing AuditLogCleanupManager >2014-08-07 14:27:12,942 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (MSC service thread 1-1) Setting audit cleanup manager to run at: 35 35 3 * * ? >2014-08-07 14:27:12,987 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (MSC service thread 1-1) Finished initializing AuditLogCleanupManager >2014-08-07 14:27:12,988 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (MSC service thread 1-1) Start initializing CommandEntityCleanupManager >2014-08-07 14:27:12,988 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (MSC service thread 1-1) Setting command entity cleanup manager to run at: 35 35 3 * * ? >2014-08-07 14:27:12,990 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (MSC service thread 1-1) Finished initializing CommandEntityCleanupManager >2014-08-07 14:27:12,992 INFO [org.ovirt.engine.core.bll.TagsDirector] (MSC service thread 1-1) Start initializing TagsDirector >2014-08-07 14:27:13,001 INFO [org.ovirt.engine.core.bll.TagsDirector] (MSC service thread 1-1) Tag root added to tree >2014-08-07 14:27:13,014 INFO [org.ovirt.engine.core.bll.TagsDirector] (MSC service thread 1-1) Finished initializing TagsDirector >2014-08-07 14:27:13,014 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (MSC service thread 1-1) Start initializing IsoDomainListSyncronizer >2014-08-07 14:27:13,023 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (MSC service thread 1-1) Finished initializing IsoDomainListSyncronizer >2014-08-07 14:27:13,188 INFO [org.ovirt.engine.core.bll.Backend] (MSC service thread 1-1) Completed initializing handlers >2014-08-07 14:27:13,431 INFO [org.ovirt.engine.core.utils.ErrorTranslatorImpl] (MSC service thread 1-1) Start initializing ErrorTranslatorImpl >2014-08-07 14:27:13,477 WARN [org.ovirt.engine.core.utils.ErrorTranslatorImpl] (MSC service thread 1-1) Code MAC_ADDRESS_IS_IN_USE appears more than once in string table. >2014-08-07 14:27:13,480 INFO [org.ovirt.engine.core.utils.ErrorTranslatorImpl] (MSC service thread 1-1) Finished initializing ErrorTranslatorImpl >2014-08-07 14:27:13,480 INFO [org.ovirt.engine.core.utils.ErrorTranslatorImpl] (MSC service thread 1-1) Start initializing ErrorTranslatorImpl >2014-08-07 14:27:13,481 INFO [org.ovirt.engine.core.utils.ErrorTranslatorImpl] (MSC service thread 1-1) Finished initializing ErrorTranslatorImpl >2014-08-07 14:27:13,482 INFO [org.ovirt.engine.core.bll.Backend] (MSC service thread 1-1) Mark incomplete jobs as UNKNOWN >2014-08-07 14:27:13,534 INFO [org.ovirt.engine.core.bll.job.JobRepositoryCleanupManager] (MSC service thread 1-1) Start initializing JobRepositoryCleanupManager >2014-08-07 14:27:13,536 INFO [org.ovirt.engine.core.bll.job.JobRepositoryCleanupManager] (MSC service thread 1-1) Finished initializing JobRepositoryCleanupManager >2014-08-07 14:27:13,537 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (MSC service thread 1-1) Start initializing AutoRecoveryManager >2014-08-07 14:27:13,541 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (MSC service thread 1-1) Finished initializing AutoRecoveryManager >2014-08-07 14:27:13,546 INFO [org.ovirt.engine.core.dal.job.ExecutionMessageDirector] (MSC service thread 1-1) Start initializing ExecutionMessageDirector >2014-08-07 14:27:13,553 INFO [org.ovirt.engine.core.dal.job.ExecutionMessageDirector] (MSC service thread 1-1) Finished initializing ExecutionMessageDirector >2014-08-07 14:27:13,617 INFO [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (MSC service thread 1-1) Start of enabled extensions list >2014-08-07 14:27:13,618 INFO [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (MSC service thread 1-1) Instance name: 'builtin-authn-internal', Extension name: 'Internal Authn (Built-in)', Version: 'N/A', Notes: '', License: 'ASL 2.0', Home: 'http://www.ovirt.org', Author 'The oVirt Project', Build interface Version: '0', File: 'N/A', Initialized: 'true' >2014-08-07 14:27:13,619 INFO [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (MSC service thread 1-1) Instance name: 'internal', Extension name: 'Internal Authz (Built-in)', Version: 'N/A', Notes: '', License: 'ASL 2.0', Home: 'http://www.ovirt.org', Author 'The oVirt Project', Build interface Version: '0', File: 'N/A', Initialized: 'true' >2014-08-07 14:27:13,620 INFO [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (MSC service thread 1-1) End of enabled extensions list >2014-08-07 14:27:13,789 INFO [org.ovirt.engine.core.bll.aaa.DbUserCacheManager] (MSC service thread 1-4) Start initializing DbUserCacheManager >2014-08-07 14:27:13,805 INFO [org.ovirt.engine.core.bll.aaa.DbUserCacheManager] (MSC service thread 1-4) Finished initializing DbUserCacheManager >2014-08-07 14:27:13,847 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (MSC service thread 1-4) Initialization of AsyncTaskManager completed successfully. >2014-08-07 14:27:13,852 INFO [org.ovirt.engine.core.vdsbroker.ResourceManager] (MSC service thread 1-4) Start initializing ResourceManager >2014-08-07 14:27:13,889 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-1) Principal admin::internal synchronized >2014-08-07 14:27:13,950 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (MSC service thread 1-4) Entered VdsManager constructor >2014-08-07 14:27:14,016 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (MSC service thread 1-4) Initialize vdsBroker (10.34.62.208,54,321) >2014-08-07 14:27:14,361 INFO [org.ovirt.engine.core.vdsbroker.ResourceManager] (MSC service thread 1-4) VDS 17f7fe56-5f32-40ef-b098-611b84484e0b was added to the Resource Manager >2014-08-07 14:27:14,451 INFO [org.ovirt.engine.core.vdsbroker.ResourceManager] (MSC service thread 1-4) Finished initializing ResourceManager >2014-08-07 14:27:14,464 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (MSC service thread 1-4) Initialization of OvfDataUpdater completed successfully. >2014-08-07 14:27:14,465 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (MSC service thread 1-4) Start scheduling to enable vds load balancer >2014-08-07 14:27:14,471 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (MSC service thread 1-4) Finished scheduling to enable vds load balancer >2014-08-07 14:27:14,472 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (MSC service thread 1-4) Start HA Reservation check >2014-08-07 14:27:14,474 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (MSC service thread 1-4) Finished HA Reservation check >2014-08-07 14:27:14,486 INFO [org.ovirt.engine.core.bll.network.MacPoolManagerRanges] (org.ovirt.thread.pool-8-thread-1) Start initializing MacPoolManagerRanges >2014-08-07 14:27:14,504 INFO [org.ovirt.engine.core.bll.InitBackendServicesOnStartupBean] (MSC service thread 1-4) Init VM custom properties utilities >2014-08-07 14:27:14,505 INFO [org.ovirt.engine.core.bll.InitBackendServicesOnStartupBean] (MSC service thread 1-4) Init device custom properties utilities >2014-08-07 14:27:14,514 INFO [org.ovirt.engine.core.bll.network.MacPoolManagerRanges] (org.ovirt.thread.pool-8-thread-1) Finished initializing. Available MACs in pool: 252 >2014-08-07 14:27:14,520 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (MSC service thread 1-4) Initializing Scheduling manager >2014-08-07 14:27:14,591 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (MSC service thread 1-4) External scheduler disabled, discovery skipped >2014-08-07 14:27:14,591 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (MSC service thread 1-4) Initialized Scheduling manager >2014-08-07 14:27:14,592 INFO [org.ovirt.engine.core.bll.dwh.DwhHeartBeat] (MSC service thread 1-4) Initializing DWH Heart Beat >2014-08-07 14:27:14,608 INFO [org.ovirt.engine.core.bll.dwh.DwhHeartBeat] (MSC service thread 1-4) DWH Heart Beat initialized >2014-08-07 14:27:15,079 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-2) Running command: LoginUserCommand internal: false. >2014-08-07 14:27:15,084 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-2) Cannot convert the string Host ${VdsName} is not responding. It will stay in Connecting state for a grace period of ${Seconds} seconds and after that an attempt to fence the host will be issued. to AuditLogType, the key does not exist in the AuditLogType declared types >2014-08-07 14:27:15,088 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-2) AuditLogType: UNASSIGNED not exist in string table >2014-08-07 14:27:15,089 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-2) AuditLogType: VDS_AUTO_FENCE_STATUS not exist in string table >2014-08-07 14:27:15,089 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-2) AuditLogType: VDS_AUTO_FENCE_STATUS_FAILED not exist in string table >2014-08-07 14:27:15,090 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-2) AuditLogType: VDS_AUTO_FENCE_FAILED_CALL_FENCE_SPM not exist in string table >2014-08-07 14:27:15,090 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-2) AuditLogType: VDS_HIGH_NETWORK_USE not exist in string table >2014-08-07 14:27:15,091 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-2) AuditLogType: USER_FAILED_REMOVE_VM not exist in string table >2014-08-07 14:27:15,092 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-2) AuditLogType: USER_RUN_UNLOCK_ENTITY_SCRIPT not exist in string table >2014-08-07 14:27:15,092 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-2) AuditLogType: VDS_NETWORK_MTU_DIFFER_FROM_LOGICAL_NETWORK not exist in string table >2014-08-07 14:27:15,093 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-2) AuditLogType: VDS_HOST_IN_CONNECTING_STATE not exist in string table >2014-08-07 14:27:15,094 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-2) AuditLogType: STORAGE_ACTIVATE_ASYNC not exist in string table >2014-08-07 14:27:15,094 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-2) AuditLogType: DWH_STOPPED not exist in string table >2014-08-07 14:27:15,095 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-2) AuditLogType: DWH_STARTED not exist in string table >2014-08-07 14:27:15,095 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-2) AuditLogType: DWH_ERROR not exist in string table >2014-08-07 14:27:15,096 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-2) AuditLogType: USER_REMOVE_AUDIT_LOG not exist in string table >2014-08-07 14:27:15,097 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-2) AuditLogType: USER_REMOVE_AUDIT_LOG_FAILED not exist in string table >2014-08-07 14:27:15,097 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-2) AuditLogType: USER_CLEAR_ALL_DISMISSED_AUDIT_LOG not exist in string table >2014-08-07 14:27:15,098 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-2) AuditLogType: USER_CLEAR_ALL_DISMISSED_AUDIT_LOG_FAILED not exist in string table >2014-08-07 14:27:15,300 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-2) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-08-07 14:27:18,185 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-6) START, GetHardwareInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]), log id: b5b109f >2014-08-07 14:27:18,247 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-6) FINISH, GetHardwareInfoVDSCommand, log id: b5b109f >2014-08-07 14:27:18,510 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-6) Initializing Host: rack >2014-08-07 14:27:18,532 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-6) [6dc144af] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS >2014-08-07 14:27:22,298 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-4) Running command: LoginUserCommand internal: false. >2014-08-07 14:27:22,309 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-4) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-08-07 14:27:24,522 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-20) [74f60fa4] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-08-07 14:27:24,573 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-20) [74f60fa4] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Active to Unknown, reason : null >2014-08-07 14:27:24,585 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-20) [74f60fa4] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Active to Unknown, reason : null >2014-08-07 14:27:24,756 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-20) [74f60fa4] Correlation ID: 74f60fa4, Call Stack: null, Custom Event ID: -1, Message: Invalid status on Data Center Default. Setting status to Non Responsive. >2014-08-07 14:27:24,983 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-20) [74f60fa4] hostFromVds::selectedVds - rack, spmStatus SPM, storage pool Default >2014-08-07 14:27:25,007 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-20) [74f60fa4] Initialize Irs proxy from vds: 10.34.62.208 >2014-08-07 14:27:25,071 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-20) [74f60fa4] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208). >2014-08-07 14:27:25,125 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-16) [74f60fa4] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: a458c7d >2014-08-07 14:27:25,181 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-16) [74f60fa4] -- executeIrsBrokerCommand: Attempting on storage pool 00000002-0002-0002-0002-0000000001d9 >2014-08-07 14:27:25,193 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-16) [74f60fa4] START, HSMGetAllTasksInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 6d5cd523 >2014-08-07 14:27:25,206 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-16) [74f60fa4] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 6d5cd523 >2014-08-07 14:27:25,208 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-16) [74f60fa4] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: a458c7d >2014-08-07 14:27:25,209 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (org.ovirt.thread.pool-8-thread-16) [74f60fa4] Discovered no tasks on Storage Pool Default >2014-08-07 14:27:40,203 INFO [org.ovirt.engine.core.bll.aaa.LoginAdminUserCommand] (ajp--127.0.0.1-8702-5) Running command: LoginAdminUserCommand internal: false. >2014-08-07 14:27:40,223 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-5) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-08-07 14:28:52,393 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Loaded file "/usr/share/ovirt-engine/services/ovirt-engine/ovirt-engine.conf". >2014-08-07 14:28:52,394 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) The file "/etc/ovirt-engine/engine.conf" doesn't exist or isn't readable. Will return an empty set of properties. >2014-08-07 14:28:52,396 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Loaded file "/etc/ovirt-engine/engine.conf.d/10-setup-database.conf". >2014-08-07 14:28:52,397 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Loaded file "/etc/ovirt-engine/engine.conf.d/10-setup-jboss.conf". >2014-08-07 14:28:52,417 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Loaded file "/etc/ovirt-engine/engine.conf.d/10-setup-pki.conf". >2014-08-07 14:28:52,418 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Loaded file "/etc/ovirt-engine/engine.conf.d/10-setup-protocols.conf". >2014-08-07 14:28:52,419 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_AJP_ENABLED" is "true". >2014-08-07 14:28:52,419 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_AJP_PORT" is "8702". >2014-08-07 14:28:52,420 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_APPS" is "engine.ear restapi.war legacy_restapi.war". >2014-08-07 14:28:52,420 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_CACHE" is "/var/cache/ovirt-engine". >2014-08-07 14:28:52,426 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_DB_CHECK_INTERVAL" is "1000". >2014-08-07 14:28:52,427 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_DB_CONNECTION_TIMEOUT" is "300000". >2014-08-07 14:28:52,427 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_DB_DATABASE" is "engine". >2014-08-07 14:28:52,428 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_DB_DRIVER" is "org.postgresql.Driver". >2014-08-07 14:28:52,428 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_DB_HOST" is "localhost". >2014-08-07 14:28:52,429 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_DB_MAX_CONNECTIONS" is "100". >2014-08-07 14:28:52,429 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_DB_MIN_CONNECTIONS" is "1". >2014-08-07 14:28:52,430 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_DB_PASSWORD" is "***". >2014-08-07 14:28:52,430 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_DB_PORT" is "5432". >2014-08-07 14:28:52,432 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_DB_SECURED" is "False". >2014-08-07 14:28:52,432 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_DB_SECURED_VALIDATION" is "False". >2014-08-07 14:28:52,433 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_DB_URL" is "jdbc:postgresql://localhost:5432/engine?sslfactory=org.postgresql.ssl.NonValidatingFactory". >2014-08-07 14:28:52,434 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_DB_USER" is "engine". >2014-08-07 14:28:52,434 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_DEBUG_ADDRESS" is "". >2014-08-07 14:28:52,435 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_DOC" is "/usr/share/doc/ovirt-engine". >2014-08-07 14:28:52,436 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_ETC" is "/etc/ovirt-engine". >2014-08-07 14:28:52,436 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_EXTENSION_PATH" is "/usr/share/ovirt-engine/extensions.d:/etc/ovirt-engine/extensions.d". >2014-08-07 14:28:52,437 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_FQDN" is "pb-rh35.rhev.lab.eng.brq.redhat.com". >2014-08-07 14:28:52,437 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_GROUP" is "ovirt". >2014-08-07 14:28:52,438 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_HEAP_MAX" is "1g". >2014-08-07 14:28:52,438 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_HEAP_MIN" is "1g". >2014-08-07 14:28:52,439 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_HTTPS_ENABLED" is "false". >2014-08-07 14:28:52,439 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_HTTPS_PORT" is "None". >2014-08-07 14:28:52,440 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_HTTPS_PROTOCOLS" is "SSLv3,TLSv1,TLSv1.1,TLSv1.2". >2014-08-07 14:28:52,440 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_HTTP_ENABLED" is "false". >2014-08-07 14:28:52,441 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_HTTP_PORT" is "None". >2014-08-07 14:28:52,441 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_JAVA_MODULEPATH" is "/usr/share/ovirt-engine/modules". >2014-08-07 14:28:52,443 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_JVM_ARGS" is " -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath="/var/log/ovirt-engine/dump"". >2014-08-07 14:28:52,443 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_LOG" is "/var/log/ovirt-engine". >2014-08-07 14:28:52,444 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_LOG_TO_CONSOLE" is "false". >2014-08-07 14:28:52,444 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_MANUAL" is "/usr/share/ovirt-engine/manual". >2014-08-07 14:28:52,445 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_PERM_MAX" is "256m". >2014-08-07 14:28:52,446 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_PERM_MIN" is "256m". >2014-08-07 14:28:52,446 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_PKI" is "/etc/pki/ovirt-engine". >2014-08-07 14:28:52,446 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_PKI_CA" is "/etc/pki/ovirt-engine/ca.pem". >2014-08-07 14:28:52,447 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_PKI_ENGINE_CERT" is "/etc/pki/ovirt-engine/certs/engine.cer". >2014-08-07 14:28:52,448 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_PKI_ENGINE_STORE" is "/etc/pki/ovirt-engine/keys/engine.p12". >2014-08-07 14:28:52,448 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_PKI_ENGINE_STORE_ALIAS" is "1". >2014-08-07 14:28:52,449 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_PKI_ENGINE_STORE_PASSWORD" is "***". >2014-08-07 14:28:52,450 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_PKI_TRUST_STORE" is "/etc/pki/ovirt-engine/.truststore". >2014-08-07 14:28:52,451 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_PKI_TRUST_STORE_PASSWORD" is "***". >2014-08-07 14:28:52,451 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_PROPERTIES" is " jsse.enableSNIExtension=false "java.security.krb5.conf=/etc/ovirt-engine/krb5.conf"". >2014-08-07 14:28:52,452 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_PROXY_ENABLED" is "true". >2014-08-07 14:28:52,453 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_PROXY_HTTPS_PORT" is "443". >2014-08-07 14:28:52,453 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_PROXY_HTTP_PORT" is "80". >2014-08-07 14:28:52,453 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_REPORTS_UI" is "/var/lib/ovirt-engine/reports.xml". >2014-08-07 14:28:52,454 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_STOP_INTERVAL" is "1". >2014-08-07 14:28:52,454 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_STOP_TIME" is "10". >2014-08-07 14:28:52,455 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_TMP" is "/var/tmp/ovirt-engine". >2014-08-07 14:28:52,455 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_UP_MARK" is "/var/lib/ovirt-engine/engine.up". >2014-08-07 14:28:52,456 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_URI" is "/ovirt-engine". >2014-08-07 14:28:52,456 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_USER" is "ovirt". >2014-08-07 14:28:52,456 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_USR" is "/usr/share/ovirt-engine". >2014-08-07 14:28:52,457 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_VAR" is "/var/lib/ovirt-engine". >2014-08-07 14:28:52,457 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_VERBOSE_GC" is "false". >2014-08-07 14:28:52,458 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "JBOSS_HOME" is "/usr/share/ovirt-engine-jboss-as". >2014-08-07 14:28:52,459 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "JBOSS_RUNTIME" is "/var/lib/ovirt-engine/jboss_runtime". >2014-08-07 14:28:52,459 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "SENSITIVE_KEYS" is ",ENGINE_DB_PASSWORD,ENGINE_PKI_TRUST_STORE_PASSWORD,ENGINE_PKI_ENGINE_STORE_PASSWORD". >2014-08-07 14:28:53,219 INFO [org.ovirt.engine.core.bll.Backend] (MSC service thread 1-1) Start initializing Backend >2014-08-07 14:28:53,623 INFO [org.ovirt.engine.core.utils.osinfo.OsInfoPreferencesLoader] (MSC service thread 1-1) Loaded file /etc/ovirt-engine/osinfo.conf.d/00-defaults.properties >2014-08-07 14:28:53,712 INFO [org.ovirt.engine.core.bll.Backend] (MSC service thread 1-1) Running ovirt-engine 3.5.0-0.0.master.20140804172041.git23b558e.el6 >2014-08-07 14:28:53,712 INFO [org.ovirt.engine.core.bll.CpuFlagsManagerHandler] (MSC service thread 1-1) Start initializing dictionaries >2014-08-07 14:28:53,715 INFO [org.ovirt.engine.core.bll.CpuFlagsManagerHandler] (MSC service thread 1-1) Finished initializing dictionaries >2014-08-07 14:28:53,715 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (MSC service thread 1-1) Start initializing AuditLogCleanupManager >2014-08-07 14:28:53,716 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (MSC service thread 1-1) Setting audit cleanup manager to run at: 35 35 3 * * ? >2014-08-07 14:28:53,736 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (MSC service thread 1-1) Finished initializing AuditLogCleanupManager >2014-08-07 14:28:53,737 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (MSC service thread 1-1) Start initializing CommandEntityCleanupManager >2014-08-07 14:28:53,737 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (MSC service thread 1-1) Setting command entity cleanup manager to run at: 35 35 3 * * ? >2014-08-07 14:28:53,740 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (MSC service thread 1-1) Finished initializing CommandEntityCleanupManager >2014-08-07 14:28:53,742 INFO [org.ovirt.engine.core.bll.TagsDirector] (MSC service thread 1-1) Start initializing TagsDirector >2014-08-07 14:28:53,751 INFO [org.ovirt.engine.core.bll.TagsDirector] (MSC service thread 1-1) Tag root added to tree >2014-08-07 14:28:53,759 INFO [org.ovirt.engine.core.bll.TagsDirector] (MSC service thread 1-1) Finished initializing TagsDirector >2014-08-07 14:28:53,759 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (MSC service thread 1-1) Start initializing IsoDomainListSyncronizer >2014-08-07 14:28:53,766 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (MSC service thread 1-1) Finished initializing IsoDomainListSyncronizer >2014-08-07 14:28:53,835 INFO [org.ovirt.engine.core.bll.Backend] (MSC service thread 1-1) Completed initializing handlers >2014-08-07 14:28:53,861 INFO [org.ovirt.engine.core.utils.ErrorTranslatorImpl] (MSC service thread 1-1) Start initializing ErrorTranslatorImpl >2014-08-07 14:28:53,872 WARN [org.ovirt.engine.core.utils.ErrorTranslatorImpl] (MSC service thread 1-1) Code MAC_ADDRESS_IS_IN_USE appears more than once in string table. >2014-08-07 14:28:53,873 INFO [org.ovirt.engine.core.utils.ErrorTranslatorImpl] (MSC service thread 1-1) Finished initializing ErrorTranslatorImpl >2014-08-07 14:28:53,874 INFO [org.ovirt.engine.core.utils.ErrorTranslatorImpl] (MSC service thread 1-1) Start initializing ErrorTranslatorImpl >2014-08-07 14:28:53,875 INFO [org.ovirt.engine.core.utils.ErrorTranslatorImpl] (MSC service thread 1-1) Finished initializing ErrorTranslatorImpl >2014-08-07 14:28:53,875 INFO [org.ovirt.engine.core.bll.Backend] (MSC service thread 1-1) Mark incomplete jobs as UNKNOWN >2014-08-07 14:28:53,908 INFO [org.ovirt.engine.core.bll.job.JobRepositoryCleanupManager] (MSC service thread 1-1) Start initializing JobRepositoryCleanupManager >2014-08-07 14:28:53,909 INFO [org.ovirt.engine.core.bll.job.JobRepositoryCleanupManager] (MSC service thread 1-1) Finished initializing JobRepositoryCleanupManager >2014-08-07 14:28:53,909 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (MSC service thread 1-1) Start initializing AutoRecoveryManager >2014-08-07 14:28:53,912 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (MSC service thread 1-1) Finished initializing AutoRecoveryManager >2014-08-07 14:28:53,916 INFO [org.ovirt.engine.core.dal.job.ExecutionMessageDirector] (MSC service thread 1-1) Start initializing ExecutionMessageDirector >2014-08-07 14:28:53,922 INFO [org.ovirt.engine.core.dal.job.ExecutionMessageDirector] (MSC service thread 1-1) Finished initializing ExecutionMessageDirector >2014-08-07 14:28:53,974 INFO [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (MSC service thread 1-1) Start of enabled extensions list >2014-08-07 14:28:53,975 INFO [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (MSC service thread 1-1) Instance name: 'builtin-authn-internal', Extension name: 'Internal Authn (Built-in)', Version: 'N/A', Notes: '', License: 'ASL 2.0', Home: 'http://www.ovirt.org', Author 'The oVirt Project', Build interface Version: '0', File: 'N/A', Initialized: 'true' >2014-08-07 14:28:53,976 INFO [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (MSC service thread 1-1) Instance name: 'internal', Extension name: 'Internal Authz (Built-in)', Version: 'N/A', Notes: '', License: 'ASL 2.0', Home: 'http://www.ovirt.org', Author 'The oVirt Project', Build interface Version: '0', File: 'N/A', Initialized: 'true' >2014-08-07 14:28:53,977 INFO [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (MSC service thread 1-1) End of enabled extensions list >2014-08-07 14:28:54,081 INFO [org.ovirt.engine.core.bll.aaa.DbUserCacheManager] (MSC service thread 1-1) Start initializing DbUserCacheManager >2014-08-07 14:28:54,082 INFO [org.ovirt.engine.core.bll.aaa.DbUserCacheManager] (MSC service thread 1-1) Finished initializing DbUserCacheManager >2014-08-07 14:28:54,131 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (MSC service thread 1-1) Initialization of AsyncTaskManager completed successfully. >2014-08-07 14:28:54,134 INFO [org.ovirt.engine.core.vdsbroker.ResourceManager] (MSC service thread 1-1) Start initializing ResourceManager >2014-08-07 14:28:54,173 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-1) Principal admin::internal synchronized >2014-08-07 14:28:54,282 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (MSC service thread 1-1) Entered VdsManager constructor >2014-08-07 14:28:54,351 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (MSC service thread 1-1) Initialize vdsBroker (10.34.62.208,54,321) >2014-08-07 14:28:54,617 INFO [org.ovirt.engine.core.vdsbroker.ResourceManager] (MSC service thread 1-1) VDS 17f7fe56-5f32-40ef-b098-611b84484e0b was added to the Resource Manager >2014-08-07 14:28:54,669 INFO [org.ovirt.engine.core.vdsbroker.ResourceManager] (MSC service thread 1-1) Finished initializing ResourceManager >2014-08-07 14:28:54,678 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (MSC service thread 1-1) Initialization of OvfDataUpdater completed successfully. >2014-08-07 14:28:54,679 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (MSC service thread 1-1) Start scheduling to enable vds load balancer >2014-08-07 14:28:54,681 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (MSC service thread 1-1) Finished scheduling to enable vds load balancer >2014-08-07 14:28:54,682 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (MSC service thread 1-1) Start HA Reservation check >2014-08-07 14:28:54,683 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (MSC service thread 1-1) Finished HA Reservation check >2014-08-07 14:28:54,711 INFO [org.ovirt.engine.core.bll.network.MacPoolManagerRanges] (org.ovirt.thread.pool-8-thread-1) Start initializing MacPoolManagerRanges >2014-08-07 14:28:54,716 INFO [org.ovirt.engine.core.bll.InitBackendServicesOnStartupBean] (MSC service thread 1-1) Init VM custom properties utilities >2014-08-07 14:28:54,717 INFO [org.ovirt.engine.core.bll.InitBackendServicesOnStartupBean] (MSC service thread 1-1) Init device custom properties utilities >2014-08-07 14:28:54,728 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (MSC service thread 1-1) Initializing Scheduling manager >2014-08-07 14:28:54,745 INFO [org.ovirt.engine.core.bll.network.MacPoolManagerRanges] (org.ovirt.thread.pool-8-thread-1) Finished initializing. Available MACs in pool: 252 >2014-08-07 14:28:54,797 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (MSC service thread 1-1) External scheduler disabled, discovery skipped >2014-08-07 14:28:54,798 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (MSC service thread 1-1) Initialized Scheduling manager >2014-08-07 14:28:54,798 INFO [org.ovirt.engine.core.bll.dwh.DwhHeartBeat] (MSC service thread 1-1) Initializing DWH Heart Beat >2014-08-07 14:28:54,803 INFO [org.ovirt.engine.core.bll.dwh.DwhHeartBeat] (MSC service thread 1-1) DWH Heart Beat initialized >2014-08-07 14:28:56,030 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-3) Running command: LoginUserCommand internal: false. >2014-08-07 14:28:56,039 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-3) Cannot convert the string Host ${VdsName} is not responding. It will stay in Connecting state for a grace period of ${Seconds} seconds and after that an attempt to fence the host will be issued. to AuditLogType, the key does not exist in the AuditLogType declared types >2014-08-07 14:28:56,049 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-3) AuditLogType: UNASSIGNED not exist in string table >2014-08-07 14:28:56,050 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-3) AuditLogType: VDS_AUTO_FENCE_STATUS not exist in string table >2014-08-07 14:28:56,051 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-3) AuditLogType: VDS_AUTO_FENCE_STATUS_FAILED not exist in string table >2014-08-07 14:28:56,052 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-3) AuditLogType: VDS_AUTO_FENCE_FAILED_CALL_FENCE_SPM not exist in string table >2014-08-07 14:28:56,053 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-3) AuditLogType: VDS_HIGH_NETWORK_USE not exist in string table >2014-08-07 14:28:56,054 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-3) AuditLogType: USER_FAILED_REMOVE_VM not exist in string table >2014-08-07 14:28:56,055 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-3) AuditLogType: USER_RUN_UNLOCK_ENTITY_SCRIPT not exist in string table >2014-08-07 14:28:56,056 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-3) AuditLogType: VDS_NETWORK_MTU_DIFFER_FROM_LOGICAL_NETWORK not exist in string table >2014-08-07 14:28:56,057 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-3) AuditLogType: VDS_HOST_IN_CONNECTING_STATE not exist in string table >2014-08-07 14:28:56,058 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-3) AuditLogType: STORAGE_ACTIVATE_ASYNC not exist in string table >2014-08-07 14:28:56,059 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-3) AuditLogType: DWH_STOPPED not exist in string table >2014-08-07 14:28:56,060 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-3) AuditLogType: DWH_STARTED not exist in string table >2014-08-07 14:28:56,061 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-3) AuditLogType: DWH_ERROR not exist in string table >2014-08-07 14:28:56,062 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-3) AuditLogType: USER_REMOVE_AUDIT_LOG not exist in string table >2014-08-07 14:28:56,064 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-3) AuditLogType: USER_REMOVE_AUDIT_LOG_FAILED not exist in string table >2014-08-07 14:28:56,065 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-3) AuditLogType: USER_CLEAR_ALL_DISMISSED_AUDIT_LOG not exist in string table >2014-08-07 14:28:56,067 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-3) AuditLogType: USER_CLEAR_ALL_DISMISSED_AUDIT_LOG_FAILED not exist in string table >2014-08-07 14:28:56,340 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-3) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-08-07 14:28:58,515 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-6) START, GetHardwareInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]), log id: 69864cda >2014-08-07 14:28:58,570 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-6) FINISH, GetHardwareInfoVDSCommand, log id: 69864cda >2014-08-07 14:28:58,857 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-6) Initializing Host: rack >2014-08-07 14:28:58,878 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-6) [45bfd8e8] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS >2014-08-07 14:29:04,743 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-20) [93c60b5] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-08-07 14:29:04,754 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-20) [93c60b5] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Active to Unknown, reason : null >2014-08-07 14:29:04,769 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-20) [93c60b5] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Active to Unknown, reason : null >2014-08-07 14:29:04,814 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-20) [93c60b5] Correlation ID: 93c60b5, Call Stack: null, Custom Event ID: -1, Message: Invalid status on Data Center Default. Setting status to Non Responsive. >2014-08-07 14:29:04,987 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-20) [93c60b5] hostFromVds::selectedVds - rack, spmStatus SPM, storage pool Default >2014-08-07 14:29:05,005 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-20) [93c60b5] Initialize Irs proxy from vds: 10.34.62.208 >2014-08-07 14:29:05,027 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-20) [93c60b5] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208). >2014-08-07 14:29:05,114 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-15) [93c60b5] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: f502e86 >2014-08-07 14:29:05,172 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-15) [93c60b5] -- executeIrsBrokerCommand: Attempting on storage pool 00000002-0002-0002-0002-0000000001d9 >2014-08-07 14:29:05,185 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-15) [93c60b5] START, HSMGetAllTasksInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 5da22ddc >2014-08-07 14:29:05,196 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-15) [93c60b5] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 5da22ddc >2014-08-07 14:29:05,198 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-15) [93c60b5] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: f502e86 >2014-08-07 14:29:05,199 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (org.ovirt.thread.pool-8-thread-15) [93c60b5] Discovered no tasks on Storage Pool Default >2014-08-07 14:29:22,110 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-5) Running command: LoginUserCommand internal: false. >2014-08-07 14:29:22,115 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-5) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-08-07 14:29:40,201 INFO [org.ovirt.engine.core.bll.aaa.LoginAdminUserCommand] (ajp--127.0.0.1-8702-6) Running command: LoginAdminUserCommand internal: false. >2014-08-07 14:29:40,220 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-08-07 14:31:40,853 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Loaded file "/usr/share/ovirt-engine/services/ovirt-engine/ovirt-engine.conf". >2014-08-07 14:31:40,853 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) The file "/etc/ovirt-engine/engine.conf" doesn't exist or isn't readable. Will return an empty set of properties. >2014-08-07 14:31:40,854 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Loaded file "/etc/ovirt-engine/engine.conf.d/10-setup-database.conf". >2014-08-07 14:31:40,855 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Loaded file "/etc/ovirt-engine/engine.conf.d/10-setup-jboss.conf". >2014-08-07 14:31:41,066 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Loaded file "/etc/ovirt-engine/engine.conf.d/10-setup-pki.conf". >2014-08-07 14:31:41,066 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Loaded file "/etc/ovirt-engine/engine.conf.d/10-setup-protocols.conf". >2014-08-07 14:31:41,067 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_AJP_ENABLED" is "true". >2014-08-07 14:31:41,068 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_AJP_PORT" is "8702". >2014-08-07 14:31:41,068 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_APPS" is "engine.ear restapi.war legacy_restapi.war". >2014-08-07 14:31:41,069 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_CACHE" is "/var/cache/ovirt-engine". >2014-08-07 14:31:41,069 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_DB_CHECK_INTERVAL" is "1000". >2014-08-07 14:31:41,077 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_DB_CONNECTION_TIMEOUT" is "300000". >2014-08-07 14:31:41,077 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_DB_DATABASE" is "engine". >2014-08-07 14:31:41,078 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_DB_DRIVER" is "org.postgresql.Driver". >2014-08-07 14:31:41,078 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_DB_HOST" is "localhost". >2014-08-07 14:31:41,079 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_DB_MAX_CONNECTIONS" is "100". >2014-08-07 14:31:41,079 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_DB_MIN_CONNECTIONS" is "1". >2014-08-07 14:31:41,080 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_DB_PASSWORD" is "***". >2014-08-07 14:31:41,080 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_DB_PORT" is "5432". >2014-08-07 14:31:41,081 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_DB_SECURED" is "False". >2014-08-07 14:31:41,081 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_DB_SECURED_VALIDATION" is "False". >2014-08-07 14:31:41,082 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_DB_URL" is "jdbc:postgresql://localhost:5432/engine?sslfactory=org.postgresql.ssl.NonValidatingFactory". >2014-08-07 14:31:41,082 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_DB_USER" is "engine". >2014-08-07 14:31:41,092 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_DEBUG_ADDRESS" is "". >2014-08-07 14:31:41,093 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_DOC" is "/usr/share/doc/ovirt-engine". >2014-08-07 14:31:41,093 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_ETC" is "/etc/ovirt-engine". >2014-08-07 14:31:41,094 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_EXTENSION_PATH" is "/usr/share/ovirt-engine/extensions.d:/etc/ovirt-engine/extensions.d". >2014-08-07 14:31:41,094 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_FQDN" is "pb-rh35.rhev.lab.eng.brq.redhat.com". >2014-08-07 14:31:41,095 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_GROUP" is "ovirt". >2014-08-07 14:31:41,095 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_HEAP_MAX" is "1g". >2014-08-07 14:31:41,096 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_HEAP_MIN" is "1g". >2014-08-07 14:31:41,096 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_HTTPS_ENABLED" is "false". >2014-08-07 14:31:41,096 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_HTTPS_PORT" is "None". >2014-08-07 14:31:41,097 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_HTTPS_PROTOCOLS" is "SSLv3,TLSv1,TLSv1.1,TLSv1.2". >2014-08-07 14:31:41,097 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_HTTP_ENABLED" is "false". >2014-08-07 14:31:41,100 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_HTTP_PORT" is "None". >2014-08-07 14:31:41,101 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_JAVA_MODULEPATH" is "/usr/share/ovirt-engine/modules". >2014-08-07 14:31:41,101 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_JVM_ARGS" is " -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath="/var/log/ovirt-engine/dump"". >2014-08-07 14:31:41,102 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_LOG" is "/var/log/ovirt-engine". >2014-08-07 14:31:41,103 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_LOG_TO_CONSOLE" is "false". >2014-08-07 14:31:41,126 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_MANUAL" is "/usr/share/ovirt-engine/manual". >2014-08-07 14:31:41,126 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_PERM_MAX" is "256m". >2014-08-07 14:31:41,127 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_PERM_MIN" is "256m". >2014-08-07 14:31:41,127 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_PKI" is "/etc/pki/ovirt-engine". >2014-08-07 14:31:41,128 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_PKI_CA" is "/etc/pki/ovirt-engine/ca.pem". >2014-08-07 14:31:41,139 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_PKI_ENGINE_CERT" is "/etc/pki/ovirt-engine/certs/engine.cer". >2014-08-07 14:31:41,140 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_PKI_ENGINE_STORE" is "/etc/pki/ovirt-engine/keys/engine.p12". >2014-08-07 14:31:41,140 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_PKI_ENGINE_STORE_ALIAS" is "1". >2014-08-07 14:31:41,141 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_PKI_ENGINE_STORE_PASSWORD" is "***". >2014-08-07 14:31:41,141 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_PKI_TRUST_STORE" is "/etc/pki/ovirt-engine/.truststore". >2014-08-07 14:31:41,157 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_PKI_TRUST_STORE_PASSWORD" is "***". >2014-08-07 14:31:41,157 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_PROPERTIES" is " jsse.enableSNIExtension=false "java.security.krb5.conf=/etc/ovirt-engine/krb5.conf"". >2014-08-07 14:31:41,158 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_PROXY_ENABLED" is "true". >2014-08-07 14:31:41,158 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_PROXY_HTTPS_PORT" is "443". >2014-08-07 14:31:41,159 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_PROXY_HTTP_PORT" is "80". >2014-08-07 14:31:41,159 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_REPORTS_UI" is "/var/lib/ovirt-engine/reports.xml". >2014-08-07 14:31:41,160 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_STOP_INTERVAL" is "1". >2014-08-07 14:31:41,160 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_STOP_TIME" is "10". >2014-08-07 14:31:41,161 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_TMP" is "/var/tmp/ovirt-engine". >2014-08-07 14:31:41,167 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_UP_MARK" is "/var/lib/ovirt-engine/engine.up". >2014-08-07 14:31:41,168 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_URI" is "/ovirt-engine". >2014-08-07 14:31:41,168 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_USER" is "ovirt". >2014-08-07 14:31:41,168 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_USR" is "/usr/share/ovirt-engine". >2014-08-07 14:31:41,169 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_VAR" is "/var/lib/ovirt-engine". >2014-08-07 14:31:41,169 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_VERBOSE_GC" is "false". >2014-08-07 14:31:41,170 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "JBOSS_HOME" is "/usr/share/ovirt-engine-jboss-as". >2014-08-07 14:31:41,170 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "JBOSS_RUNTIME" is "/var/lib/ovirt-engine/jboss_runtime". >2014-08-07 14:31:41,186 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "SENSITIVE_KEYS" is ",ENGINE_DB_PASSWORD,ENGINE_PKI_TRUST_STORE_PASSWORD,ENGINE_PKI_ENGINE_STORE_PASSWORD". >2014-08-07 14:31:41,587 ERROR [org.ovirt.engine.core.aaa.filters.BasicAuthenticationFilter] (ajp--127.0.0.1-8702-9) Error in obtaining profile null >2014-08-07 14:31:42,174 INFO [org.ovirt.engine.core.bll.Backend] (MSC service thread 1-1) Start initializing Backend >2014-08-07 14:31:42,243 ERROR [org.ovirt.engine.core.aaa.filters.BasicAuthenticationFilter] (ajp--127.0.0.1-8702-2) Error in obtaining profile null >2014-08-07 14:31:43,295 ERROR [org.ovirt.engine.core.aaa.filters.BasicAuthenticationFilter] (ajp--127.0.0.1-8702-8) Error in obtaining profile null >2014-08-07 14:31:43,305 INFO [org.ovirt.engine.core.utils.osinfo.OsInfoPreferencesLoader] (MSC service thread 1-1) Loaded file /etc/ovirt-engine/osinfo.conf.d/00-defaults.properties >2014-08-07 14:31:43,311 ERROR [org.ovirt.engine.core.aaa.filters.BasicAuthenticationFilter] (ajp--127.0.0.1-8702-9) Error in obtaining profile null >2014-08-07 14:31:43,332 ERROR [org.ovirt.engine.core.aaa.filters.BasicAuthenticationFilter] (ajp--127.0.0.1-8702-6) Error in obtaining profile null >2014-08-07 14:31:43,351 ERROR [org.ovirt.engine.core.aaa.filters.BasicAuthenticationFilter] (ajp--127.0.0.1-8702-10) Error in obtaining profile null >2014-08-07 14:31:43,377 ERROR [org.ovirt.engine.core.aaa.filters.BasicAuthenticationFilter] (ajp--127.0.0.1-8702-11) Error in obtaining profile null >2014-08-07 14:31:43,399 ERROR [org.ovirt.engine.core.aaa.filters.BasicAuthenticationFilter] (ajp--127.0.0.1-8702-1) Error in obtaining profile null >2014-08-07 14:31:43,418 ERROR [org.ovirt.engine.core.aaa.filters.BasicAuthenticationFilter] (ajp--127.0.0.1-8702-3) Error in obtaining profile null >2014-08-07 14:31:43,444 ERROR [org.ovirt.engine.core.aaa.filters.BasicAuthenticationFilter] (ajp--127.0.0.1-8702-4) Error in obtaining profile null >2014-08-07 14:31:43,463 ERROR [org.ovirt.engine.core.aaa.filters.BasicAuthenticationFilter] (ajp--127.0.0.1-8702-5) Error in obtaining profile null >2014-08-07 14:31:43,482 ERROR [org.ovirt.engine.core.aaa.filters.BasicAuthenticationFilter] (ajp--127.0.0.1-8702-7) Error in obtaining profile null >2014-08-07 14:31:43,487 INFO [org.ovirt.engine.core.bll.Backend] (MSC service thread 1-1) Running ovirt-engine 3.5.0-0.0.master.20140804172041.git23b558e.el6 >2014-08-07 14:31:43,488 INFO [org.ovirt.engine.core.bll.CpuFlagsManagerHandler] (MSC service thread 1-1) Start initializing dictionaries >2014-08-07 14:31:43,494 INFO [org.ovirt.engine.core.bll.CpuFlagsManagerHandler] (MSC service thread 1-1) Finished initializing dictionaries >2014-08-07 14:31:43,494 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (MSC service thread 1-1) Start initializing AuditLogCleanupManager >2014-08-07 14:31:43,495 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (MSC service thread 1-1) Setting audit cleanup manager to run at: 35 35 3 * * ? >2014-08-07 14:31:43,505 ERROR [org.ovirt.engine.core.aaa.filters.BasicAuthenticationFilter] (ajp--127.0.0.1-8702-8) Error in obtaining profile null >2014-08-07 14:31:43,536 ERROR [org.ovirt.engine.core.aaa.filters.BasicAuthenticationFilter] (ajp--127.0.0.1-8702-9) Error in obtaining profile null >2014-08-07 14:31:43,548 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (MSC service thread 1-1) Finished initializing AuditLogCleanupManager >2014-08-07 14:31:43,548 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (MSC service thread 1-1) Start initializing CommandEntityCleanupManager >2014-08-07 14:31:43,549 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (MSC service thread 1-1) Setting command entity cleanup manager to run at: 35 35 3 * * ? >2014-08-07 14:31:43,554 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (MSC service thread 1-1) Finished initializing CommandEntityCleanupManager >2014-08-07 14:31:43,555 ERROR [org.ovirt.engine.core.aaa.filters.BasicAuthenticationFilter] (ajp--127.0.0.1-8702-6) Error in obtaining profile null >2014-08-07 14:31:43,559 INFO [org.ovirt.engine.core.bll.TagsDirector] (MSC service thread 1-1) Start initializing TagsDirector >2014-08-07 14:31:43,581 ERROR [org.ovirt.engine.core.aaa.filters.BasicAuthenticationFilter] (ajp--127.0.0.1-8702-10) Error in obtaining profile null >2014-08-07 14:31:43,582 INFO [org.ovirt.engine.core.bll.TagsDirector] (MSC service thread 1-1) Tag root added to tree >2014-08-07 14:31:43,596 INFO [org.ovirt.engine.core.bll.TagsDirector] (MSC service thread 1-1) Finished initializing TagsDirector >2014-08-07 14:31:43,596 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (MSC service thread 1-1) Start initializing IsoDomainListSyncronizer >2014-08-07 14:31:43,607 ERROR [org.ovirt.engine.core.aaa.filters.BasicAuthenticationFilter] (ajp--127.0.0.1-8702-11) Error in obtaining profile null >2014-08-07 14:31:43,608 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (MSC service thread 1-1) Finished initializing IsoDomainListSyncronizer >2014-08-07 14:31:43,634 ERROR [org.ovirt.engine.core.aaa.filters.BasicAuthenticationFilter] (ajp--127.0.0.1-8702-1) Error in obtaining profile null >2014-08-07 14:31:43,652 ERROR [org.ovirt.engine.core.aaa.filters.BasicAuthenticationFilter] (ajp--127.0.0.1-8702-3) Error in obtaining profile null >2014-08-07 14:31:43,678 ERROR [org.ovirt.engine.core.aaa.filters.BasicAuthenticationFilter] (ajp--127.0.0.1-8702-4) Error in obtaining profile null >2014-08-07 14:31:43,698 ERROR [org.ovirt.engine.core.aaa.filters.BasicAuthenticationFilter] (ajp--127.0.0.1-8702-5) Error in obtaining profile null >2014-08-07 14:31:43,717 ERROR [org.ovirt.engine.core.aaa.filters.BasicAuthenticationFilter] (ajp--127.0.0.1-8702-7) Error in obtaining profile null >2014-08-07 14:31:43,736 ERROR [org.ovirt.engine.core.aaa.filters.BasicAuthenticationFilter] (ajp--127.0.0.1-8702-8) Error in obtaining profile null >2014-08-07 14:31:43,761 ERROR [org.ovirt.engine.core.aaa.filters.BasicAuthenticationFilter] (ajp--127.0.0.1-8702-9) Error in obtaining profile null >2014-08-07 14:31:43,779 ERROR [org.ovirt.engine.core.aaa.filters.BasicAuthenticationFilter] (ajp--127.0.0.1-8702-6) Error in obtaining profile null >2014-08-07 14:31:43,794 INFO [org.ovirt.engine.core.bll.Backend] (MSC service thread 1-1) Completed initializing handlers >2014-08-07 14:31:43,799 ERROR [org.ovirt.engine.core.aaa.filters.BasicAuthenticationFilter] (ajp--127.0.0.1-8702-10) Error in obtaining profile null >2014-08-07 14:31:43,819 ERROR [org.ovirt.engine.core.aaa.filters.BasicAuthenticationFilter] (ajp--127.0.0.1-8702-11) Error in obtaining profile null >2014-08-07 14:31:43,846 INFO [org.ovirt.engine.core.utils.ErrorTranslatorImpl] (MSC service thread 1-1) Start initializing ErrorTranslatorImpl >2014-08-07 14:31:43,850 ERROR [org.ovirt.engine.core.aaa.filters.BasicAuthenticationFilter] (ajp--127.0.0.1-8702-1) Error in obtaining profile null >2014-08-07 14:31:43,872 ERROR [org.ovirt.engine.core.aaa.filters.BasicAuthenticationFilter] (ajp--127.0.0.1-8702-3) Error in obtaining profile null >2014-08-07 14:31:43,878 WARN [org.ovirt.engine.core.utils.ErrorTranslatorImpl] (MSC service thread 1-1) Code MAC_ADDRESS_IS_IN_USE appears more than once in string table. >2014-08-07 14:31:43,879 INFO [org.ovirt.engine.core.utils.ErrorTranslatorImpl] (MSC service thread 1-1) Finished initializing ErrorTranslatorImpl >2014-08-07 14:31:43,880 INFO [org.ovirt.engine.core.utils.ErrorTranslatorImpl] (MSC service thread 1-1) Start initializing ErrorTranslatorImpl >2014-08-07 14:31:43,887 INFO [org.ovirt.engine.core.utils.ErrorTranslatorImpl] (MSC service thread 1-1) Finished initializing ErrorTranslatorImpl >2014-08-07 14:31:43,888 INFO [org.ovirt.engine.core.bll.Backend] (MSC service thread 1-1) Mark incomplete jobs as UNKNOWN >2014-08-07 14:31:43,898 ERROR [org.ovirt.engine.core.aaa.filters.BasicAuthenticationFilter] (ajp--127.0.0.1-8702-4) Error in obtaining profile null >2014-08-07 14:31:43,921 ERROR [org.ovirt.engine.core.aaa.filters.BasicAuthenticationFilter] (ajp--127.0.0.1-8702-5) Error in obtaining profile null >2014-08-07 14:31:43,944 ERROR [org.ovirt.engine.core.aaa.filters.BasicAuthenticationFilter] (ajp--127.0.0.1-8702-7) Error in obtaining profile null >2014-08-07 14:31:43,962 INFO [org.ovirt.engine.core.bll.job.JobRepositoryCleanupManager] (MSC service thread 1-1) Start initializing JobRepositoryCleanupManager >2014-08-07 14:31:43,962 ERROR [org.ovirt.engine.core.aaa.filters.BasicAuthenticationFilter] (ajp--127.0.0.1-8702-8) Error in obtaining profile null >2014-08-07 14:31:43,962 INFO [org.ovirt.engine.core.bll.job.JobRepositoryCleanupManager] (MSC service thread 1-1) Finished initializing JobRepositoryCleanupManager >2014-08-07 14:31:43,964 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (MSC service thread 1-1) Start initializing AutoRecoveryManager >2014-08-07 14:31:43,967 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (MSC service thread 1-1) Finished initializing AutoRecoveryManager >2014-08-07 14:31:43,969 INFO [org.ovirt.engine.core.dal.job.ExecutionMessageDirector] (MSC service thread 1-1) Start initializing ExecutionMessageDirector >2014-08-07 14:31:43,978 INFO [org.ovirt.engine.core.dal.job.ExecutionMessageDirector] (MSC service thread 1-1) Finished initializing ExecutionMessageDirector >2014-08-07 14:31:43,982 ERROR [org.ovirt.engine.core.aaa.filters.BasicAuthenticationFilter] (ajp--127.0.0.1-8702-9) Error in obtaining profile null >2014-08-07 14:31:44,007 ERROR [org.ovirt.engine.core.aaa.filters.BasicAuthenticationFilter] (ajp--127.0.0.1-8702-6) Error in obtaining profile null >2014-08-07 14:31:44,026 ERROR [org.ovirt.engine.core.aaa.filters.BasicAuthenticationFilter] (ajp--127.0.0.1-8702-10) Error in obtaining profile null >2014-08-07 14:31:44,045 ERROR [org.ovirt.engine.core.aaa.filters.BasicAuthenticationFilter] (ajp--127.0.0.1-8702-11) Error in obtaining profile null >2014-08-07 14:31:44,071 ERROR [org.ovirt.engine.core.aaa.filters.BasicAuthenticationFilter] (ajp--127.0.0.1-8702-1) Error in obtaining profile null >2014-08-07 14:31:44,079 INFO [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (MSC service thread 1-1) Start of enabled extensions list >2014-08-07 14:31:44,080 INFO [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (MSC service thread 1-1) Instance name: 'builtin-authn-internal', Extension name: 'Internal Authn (Built-in)', Version: 'N/A', Notes: '', License: 'ASL 2.0', Home: 'http://www.ovirt.org', Author 'The oVirt Project', Build interface Version: '0', File: 'N/A', Initialized: 'true' >2014-08-07 14:31:44,081 INFO [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (MSC service thread 1-1) Instance name: 'internal', Extension name: 'Internal Authz (Built-in)', Version: 'N/A', Notes: '', License: 'ASL 2.0', Home: 'http://www.ovirt.org', Author 'The oVirt Project', Build interface Version: '0', File: 'N/A', Initialized: 'true' >2014-08-07 14:31:44,082 INFO [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (MSC service thread 1-1) End of enabled extensions list >2014-08-07 14:31:44,230 INFO [org.ovirt.engine.core.bll.aaa.DbUserCacheManager] (MSC service thread 1-1) Start initializing DbUserCacheManager >2014-08-07 14:31:44,245 INFO [org.ovirt.engine.core.bll.aaa.DbUserCacheManager] (MSC service thread 1-1) Finished initializing DbUserCacheManager >2014-08-07 14:31:44,268 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (MSC service thread 1-1) Initialization of AsyncTaskManager completed successfully. >2014-08-07 14:31:44,276 INFO [org.ovirt.engine.core.vdsbroker.ResourceManager] (MSC service thread 1-1) Start initializing ResourceManager >2014-08-07 14:31:44,365 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-1) Principal admin::internal synchronized >2014-08-07 14:31:44,436 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (MSC service thread 1-1) Entered VdsManager constructor >2014-08-07 14:31:44,487 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-3) Running command: LoginUserCommand internal: false. >2014-08-07 14:31:44,507 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-3) Cannot convert the string Host ${VdsName} is not responding. It will stay in Connecting state for a grace period of ${Seconds} seconds and after that an attempt to fence the host will be issued. to AuditLogType, the key does not exist in the AuditLogType declared types >2014-08-07 14:31:44,512 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (MSC service thread 1-1) Initialize vdsBroker (10.34.62.208,54,321) >2014-08-07 14:31:44,529 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-3) AuditLogType: UNASSIGNED not exist in string table >2014-08-07 14:31:44,529 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-3) AuditLogType: VDS_AUTO_FENCE_STATUS not exist in string table >2014-08-07 14:31:44,533 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-3) AuditLogType: VDS_AUTO_FENCE_STATUS_FAILED not exist in string table >2014-08-07 14:31:44,533 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-3) AuditLogType: VDS_AUTO_FENCE_FAILED_CALL_FENCE_SPM not exist in string table >2014-08-07 14:31:44,534 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-3) AuditLogType: VDS_HIGH_NETWORK_USE not exist in string table >2014-08-07 14:31:44,535 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-3) AuditLogType: USER_FAILED_REMOVE_VM not exist in string table >2014-08-07 14:31:44,535 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-3) AuditLogType: USER_RUN_UNLOCK_ENTITY_SCRIPT not exist in string table >2014-08-07 14:31:44,545 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-3) AuditLogType: VDS_NETWORK_MTU_DIFFER_FROM_LOGICAL_NETWORK not exist in string table >2014-08-07 14:31:44,548 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-3) AuditLogType: VDS_HOST_IN_CONNECTING_STATE not exist in string table >2014-08-07 14:31:44,548 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-3) AuditLogType: STORAGE_ACTIVATE_ASYNC not exist in string table >2014-08-07 14:31:44,549 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-3) AuditLogType: DWH_STOPPED not exist in string table >2014-08-07 14:31:44,549 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-3) AuditLogType: DWH_STARTED not exist in string table >2014-08-07 14:31:44,550 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-3) AuditLogType: DWH_ERROR not exist in string table >2014-08-07 14:31:44,554 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-3) AuditLogType: USER_REMOVE_AUDIT_LOG not exist in string table >2014-08-07 14:31:44,555 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-3) AuditLogType: USER_REMOVE_AUDIT_LOG_FAILED not exist in string table >2014-08-07 14:31:44,556 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-3) AuditLogType: USER_CLEAR_ALL_DISMISSED_AUDIT_LOG not exist in string table >2014-08-07 14:31:44,556 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-3) AuditLogType: USER_CLEAR_ALL_DISMISSED_AUDIT_LOG_FAILED not exist in string table >2014-08-07 14:31:44,828 INFO [org.ovirt.engine.core.vdsbroker.ResourceManager] (MSC service thread 1-1) VDS 17f7fe56-5f32-40ef-b098-611b84484e0b was added to the Resource Manager >2014-08-07 14:31:44,891 INFO [org.ovirt.engine.core.vdsbroker.ResourceManager] (MSC service thread 1-1) Finished initializing ResourceManager >2014-08-07 14:31:44,904 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (MSC service thread 1-1) Initialization of OvfDataUpdater completed successfully. >2014-08-07 14:31:44,905 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (MSC service thread 1-1) Start scheduling to enable vds load balancer >2014-08-07 14:31:44,912 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (MSC service thread 1-1) Finished scheduling to enable vds load balancer >2014-08-07 14:31:44,913 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (MSC service thread 1-1) Start HA Reservation check >2014-08-07 14:31:44,917 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (MSC service thread 1-1) Finished HA Reservation check >2014-08-07 14:31:44,946 INFO [org.ovirt.engine.core.bll.InitBackendServicesOnStartupBean] (MSC service thread 1-1) Init VM custom properties utilities >2014-08-07 14:31:44,950 INFO [org.ovirt.engine.core.bll.InitBackendServicesOnStartupBean] (MSC service thread 1-1) Init device custom properties utilities >2014-08-07 14:31:44,952 INFO [org.ovirt.engine.core.bll.network.MacPoolManagerRanges] (org.ovirt.thread.pool-8-thread-2) Start initializing MacPoolManagerRanges >2014-08-07 14:31:44,986 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (MSC service thread 1-1) Initializing Scheduling manager >2014-08-07 14:31:44,995 INFO [org.ovirt.engine.core.bll.network.MacPoolManagerRanges] (org.ovirt.thread.pool-8-thread-2) Finished initializing. Available MACs in pool: 252 >2014-08-07 14:31:45,057 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (MSC service thread 1-1) External scheduler disabled, discovery skipped >2014-08-07 14:31:45,058 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (MSC service thread 1-1) Initialized Scheduling manager >2014-08-07 14:31:45,059 INFO [org.ovirt.engine.core.bll.dwh.DwhHeartBeat] (MSC service thread 1-1) Initializing DWH Heart Beat >2014-08-07 14:31:45,061 INFO [org.ovirt.engine.core.bll.dwh.DwhHeartBeat] (MSC service thread 1-1) DWH Heart Beat initialized >2014-08-07 14:31:45,210 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-3) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-08-07 14:31:45,558 INFO [org.ovirt.engine.core.bll.aaa.LoginAdminUserCommand] (ajp--127.0.0.1-8702-4) Running command: LoginAdminUserCommand internal: false. >2014-08-07 14:31:45,595 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp--127.0.0.1-8702-4) [2e5e0418] Running command: LogoutUserCommand internal: false. >2014-08-07 14:31:45,634 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-4) [2e5e0418] Correlation ID: 2e5e0418, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-08-07 14:31:46,050 INFO [org.ovirt.engine.core.bll.aaa.LoginAdminUserCommand] (ajp--127.0.0.1-8702-5) Running command: LoginAdminUserCommand internal: false. >2014-08-07 14:31:48,803 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-6) START, GetHardwareInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]), log id: 5122ceb2 >2014-08-07 14:31:48,853 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-6) FINISH, GetHardwareInfoVDSCommand, log id: 5122ceb2 >2014-08-07 14:31:49,138 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-6) Initializing Host: rack >2014-08-07 14:31:49,157 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-6) [64e470f4] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS >2014-08-07 14:31:54,958 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-20) [737e3ec1] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-08-07 14:31:54,981 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-20) [737e3ec1] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Active to Unknown, reason : null >2014-08-07 14:31:54,996 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-20) [737e3ec1] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Active to Unknown, reason : null >2014-08-07 14:31:55,041 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-20) [737e3ec1] Correlation ID: 737e3ec1, Call Stack: null, Custom Event ID: -1, Message: Invalid status on Data Center Default. Setting status to Non Responsive. >2014-08-07 14:31:55,216 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-20) [737e3ec1] hostFromVds::selectedVds - rack, spmStatus SPM, storage pool Default >2014-08-07 14:31:55,235 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-20) [737e3ec1] Initialize Irs proxy from vds: 10.34.62.208 >2014-08-07 14:31:55,276 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-20) [737e3ec1] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208). >2014-08-07 14:31:55,316 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-15) [737e3ec1] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 39c28664 >2014-08-07 14:31:55,436 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-15) [737e3ec1] -- executeIrsBrokerCommand: Attempting on storage pool 00000002-0002-0002-0002-0000000001d9 >2014-08-07 14:31:55,459 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-15) [737e3ec1] START, HSMGetAllTasksInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 6fce3e95 >2014-08-07 14:31:55,516 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-15) [737e3ec1] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 6fce3e95 >2014-08-07 14:31:55,517 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-15) [737e3ec1] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 39c28664 >2014-08-07 14:31:55,518 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (org.ovirt.thread.pool-8-thread-15) [737e3ec1] Discovered no tasks on Storage Pool Default >2014-08-07 14:36:16,507 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Loaded file "/usr/share/ovirt-engine/services/ovirt-engine/ovirt-engine.conf". >2014-08-07 14:36:16,518 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) The file "/etc/ovirt-engine/engine.conf" doesn't exist or isn't readable. Will return an empty set of properties. >2014-08-07 14:36:16,519 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Loaded file "/etc/ovirt-engine/engine.conf.d/10-setup-database.conf". >2014-08-07 14:36:16,520 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Loaded file "/etc/ovirt-engine/engine.conf.d/10-setup-jboss.conf". >2014-08-07 14:36:16,534 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Loaded file "/etc/ovirt-engine/engine.conf.d/10-setup-pki.conf". >2014-08-07 14:36:16,535 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Loaded file "/etc/ovirt-engine/engine.conf.d/10-setup-protocols.conf". >2014-08-07 14:36:16,545 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_AJP_ENABLED" is "true". >2014-08-07 14:36:16,546 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_AJP_PORT" is "8702". >2014-08-07 14:36:16,547 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_APPS" is "engine.ear restapi.war legacy_restapi.war". >2014-08-07 14:36:16,556 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_CACHE" is "/var/cache/ovirt-engine". >2014-08-07 14:36:16,557 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_DB_CHECK_INTERVAL" is "1000". >2014-08-07 14:36:16,558 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_DB_CONNECTION_TIMEOUT" is "300000". >2014-08-07 14:36:16,559 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_DB_DATABASE" is "engine". >2014-08-07 14:36:16,574 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_DB_DRIVER" is "org.postgresql.Driver". >2014-08-07 14:36:16,574 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_DB_HOST" is "localhost". >2014-08-07 14:36:16,575 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_DB_MAX_CONNECTIONS" is "100". >2014-08-07 14:36:16,576 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_DB_MIN_CONNECTIONS" is "1". >2014-08-07 14:36:16,576 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_DB_PASSWORD" is "***". >2014-08-07 14:36:16,581 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_DB_PORT" is "5432". >2014-08-07 14:36:16,582 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_DB_SECURED" is "False". >2014-08-07 14:36:16,583 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_DB_SECURED_VALIDATION" is "False". >2014-08-07 14:36:16,583 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_DB_URL" is "jdbc:postgresql://localhost:5432/engine?sslfactory=org.postgresql.ssl.NonValidatingFactory". >2014-08-07 14:36:16,584 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_DB_USER" is "engine". >2014-08-07 14:36:16,593 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_DEBUG_ADDRESS" is "". >2014-08-07 14:36:16,594 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_DOC" is "/usr/share/doc/ovirt-engine". >2014-08-07 14:36:16,595 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_ETC" is "/etc/ovirt-engine". >2014-08-07 14:36:16,595 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_EXTENSION_PATH" is "/usr/share/ovirt-engine/extensions.d:/etc/ovirt-engine/extensions.d". >2014-08-07 14:36:16,596 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_FQDN" is "pb-rh35.rhev.lab.eng.brq.redhat.com". >2014-08-07 14:36:16,602 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_GROUP" is "ovirt". >2014-08-07 14:36:16,603 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_HEAP_MAX" is "1g". >2014-08-07 14:36:16,604 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_HEAP_MIN" is "1g". >2014-08-07 14:36:16,604 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_HTTPS_ENABLED" is "false". >2014-08-07 14:36:16,605 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_HTTPS_PORT" is "None". >2014-08-07 14:36:16,606 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_HTTPS_PROTOCOLS" is "SSLv3,TLSv1,TLSv1.1,TLSv1.2". >2014-08-07 14:36:16,606 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_HTTP_ENABLED" is "false". >2014-08-07 14:36:16,607 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_HTTP_PORT" is "None". >2014-08-07 14:36:16,620 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_JAVA_MODULEPATH" is "/usr/share/ovirt-engine/modules". >2014-08-07 14:36:16,621 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_JVM_ARGS" is " -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath="/var/log/ovirt-engine/dump"". >2014-08-07 14:36:16,632 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_LOG" is "/var/log/ovirt-engine". >2014-08-07 14:36:16,632 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_LOG_TO_CONSOLE" is "false". >2014-08-07 14:36:16,633 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_MANUAL" is "/usr/share/ovirt-engine/manual". >2014-08-07 14:36:16,634 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_PERM_MAX" is "256m". >2014-08-07 14:36:16,634 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_PERM_MIN" is "256m". >2014-08-07 14:36:16,638 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_PKI" is "/etc/pki/ovirt-engine". >2014-08-07 14:36:16,638 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_PKI_CA" is "/etc/pki/ovirt-engine/ca.pem". >2014-08-07 14:36:16,639 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_PKI_ENGINE_CERT" is "/etc/pki/ovirt-engine/certs/engine.cer". >2014-08-07 14:36:16,640 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_PKI_ENGINE_STORE" is "/etc/pki/ovirt-engine/keys/engine.p12". >2014-08-07 14:36:16,650 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_PKI_ENGINE_STORE_ALIAS" is "1". >2014-08-07 14:36:16,651 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_PKI_ENGINE_STORE_PASSWORD" is "***". >2014-08-07 14:36:16,652 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_PKI_TRUST_STORE" is "/etc/pki/ovirt-engine/.truststore". >2014-08-07 14:36:16,653 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_PKI_TRUST_STORE_PASSWORD" is "***". >2014-08-07 14:36:16,653 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_PROPERTIES" is " jsse.enableSNIExtension=false "java.security.krb5.conf=/etc/ovirt-engine/krb5.conf"". >2014-08-07 14:36:16,666 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_PROXY_ENABLED" is "true". >2014-08-07 14:36:16,666 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_PROXY_HTTPS_PORT" is "443". >2014-08-07 14:36:16,667 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_PROXY_HTTP_PORT" is "80". >2014-08-07 14:36:16,668 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_REPORTS_UI" is "/var/lib/ovirt-engine/reports.xml". >2014-08-07 14:36:16,669 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_STOP_INTERVAL" is "1". >2014-08-07 14:36:16,669 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_STOP_TIME" is "10". >2014-08-07 14:36:16,670 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_TMP" is "/var/tmp/ovirt-engine". >2014-08-07 14:36:16,671 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_UP_MARK" is "/var/lib/ovirt-engine/engine.up". >2014-08-07 14:36:16,672 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_URI" is "/ovirt-engine". >2014-08-07 14:36:16,685 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_USER" is "ovirt". >2014-08-07 14:36:16,685 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_USR" is "/usr/share/ovirt-engine". >2014-08-07 14:36:16,686 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_VAR" is "/var/lib/ovirt-engine". >2014-08-07 14:36:16,687 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_VERBOSE_GC" is "false". >2014-08-07 14:36:16,688 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "JBOSS_HOME" is "/usr/share/ovirt-engine-jboss-as". >2014-08-07 14:36:16,688 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "JBOSS_RUNTIME" is "/var/lib/ovirt-engine/jboss_runtime". >2014-08-07 14:36:16,689 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "SENSITIVE_KEYS" is ",ENGINE_DB_PASSWORD,ENGINE_PKI_TRUST_STORE_PASSWORD,ENGINE_PKI_ENGINE_STORE_PASSWORD". >2014-08-07 14:36:17,675 INFO [org.ovirt.engine.core.bll.Backend] (MSC service thread 1-2) Start initializing Backend >2014-08-07 14:36:18,295 INFO [org.ovirt.engine.core.utils.osinfo.OsInfoPreferencesLoader] (MSC service thread 1-2) Loaded file /etc/ovirt-engine/osinfo.conf.d/00-defaults.properties >2014-08-07 14:36:18,418 INFO [org.ovirt.engine.core.bll.Backend] (MSC service thread 1-2) Running ovirt-engine 3.5.0-0.0.master.20140804172041.git23b558e.el6 >2014-08-07 14:36:18,418 INFO [org.ovirt.engine.core.bll.CpuFlagsManagerHandler] (MSC service thread 1-2) Start initializing dictionaries >2014-08-07 14:36:18,421 INFO [org.ovirt.engine.core.bll.CpuFlagsManagerHandler] (MSC service thread 1-2) Finished initializing dictionaries >2014-08-07 14:36:18,422 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (MSC service thread 1-2) Start initializing AuditLogCleanupManager >2014-08-07 14:36:18,422 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (MSC service thread 1-2) Setting audit cleanup manager to run at: 35 35 3 * * ? >2014-08-07 14:36:18,465 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (MSC service thread 1-2) Finished initializing AuditLogCleanupManager >2014-08-07 14:36:18,466 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (MSC service thread 1-2) Start initializing CommandEntityCleanupManager >2014-08-07 14:36:18,466 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (MSC service thread 1-2) Setting command entity cleanup manager to run at: 35 35 3 * * ? >2014-08-07 14:36:18,474 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (MSC service thread 1-2) Finished initializing CommandEntityCleanupManager >2014-08-07 14:36:18,476 INFO [org.ovirt.engine.core.bll.TagsDirector] (MSC service thread 1-2) Start initializing TagsDirector >2014-08-07 14:36:18,490 INFO [org.ovirt.engine.core.bll.TagsDirector] (MSC service thread 1-2) Tag root added to tree >2014-08-07 14:36:18,510 INFO [org.ovirt.engine.core.bll.TagsDirector] (MSC service thread 1-2) Finished initializing TagsDirector >2014-08-07 14:36:18,511 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (MSC service thread 1-2) Start initializing IsoDomainListSyncronizer >2014-08-07 14:36:18,528 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (MSC service thread 1-2) Finished initializing IsoDomainListSyncronizer >2014-08-07 14:36:18,693 INFO [org.ovirt.engine.core.bll.Backend] (MSC service thread 1-2) Completed initializing handlers >2014-08-07 14:36:18,748 INFO [org.ovirt.engine.core.utils.ErrorTranslatorImpl] (MSC service thread 1-2) Start initializing ErrorTranslatorImpl >2014-08-07 14:36:18,914 WARN [org.ovirt.engine.core.utils.ErrorTranslatorImpl] (MSC service thread 1-2) Code MAC_ADDRESS_IS_IN_USE appears more than once in string table. >2014-08-07 14:36:18,916 INFO [org.ovirt.engine.core.utils.ErrorTranslatorImpl] (MSC service thread 1-2) Finished initializing ErrorTranslatorImpl >2014-08-07 14:36:18,916 INFO [org.ovirt.engine.core.utils.ErrorTranslatorImpl] (MSC service thread 1-2) Start initializing ErrorTranslatorImpl >2014-08-07 14:36:18,917 INFO [org.ovirt.engine.core.utils.ErrorTranslatorImpl] (MSC service thread 1-2) Finished initializing ErrorTranslatorImpl >2014-08-07 14:36:18,918 INFO [org.ovirt.engine.core.bll.Backend] (MSC service thread 1-2) Mark incomplete jobs as UNKNOWN >2014-08-07 14:36:18,973 INFO [org.ovirt.engine.core.bll.job.JobRepositoryCleanupManager] (MSC service thread 1-2) Start initializing JobRepositoryCleanupManager >2014-08-07 14:36:18,974 INFO [org.ovirt.engine.core.bll.job.JobRepositoryCleanupManager] (MSC service thread 1-2) Finished initializing JobRepositoryCleanupManager >2014-08-07 14:36:18,975 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (MSC service thread 1-2) Start initializing AutoRecoveryManager >2014-08-07 14:36:18,980 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (MSC service thread 1-2) Finished initializing AutoRecoveryManager >2014-08-07 14:36:18,982 INFO [org.ovirt.engine.core.dal.job.ExecutionMessageDirector] (MSC service thread 1-2) Start initializing ExecutionMessageDirector >2014-08-07 14:36:18,994 INFO [org.ovirt.engine.core.dal.job.ExecutionMessageDirector] (MSC service thread 1-2) Finished initializing ExecutionMessageDirector >2014-08-07 14:36:19,077 INFO [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (MSC service thread 1-2) Start of enabled extensions list >2014-08-07 14:36:19,080 INFO [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (MSC service thread 1-2) Instance name: 'builtin-authn-internal', Extension name: 'Internal Authn (Built-in)', Version: 'N/A', Notes: '', License: 'ASL 2.0', Home: 'http://www.ovirt.org', Author 'The oVirt Project', Build interface Version: '0', File: 'N/A', Initialized: 'true' >2014-08-07 14:36:19,085 INFO [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (MSC service thread 1-2) Instance name: 'internal', Extension name: 'Internal Authz (Built-in)', Version: 'N/A', Notes: '', License: 'ASL 2.0', Home: 'http://www.ovirt.org', Author 'The oVirt Project', Build interface Version: '0', File: 'N/A', Initialized: 'true' >2014-08-07 14:36:19,086 INFO [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (MSC service thread 1-2) End of enabled extensions list >2014-08-07 14:36:19,197 INFO [org.ovirt.engine.core.bll.aaa.DbUserCacheManager] (MSC service thread 1-2) Start initializing DbUserCacheManager >2014-08-07 14:36:19,202 INFO [org.ovirt.engine.core.bll.aaa.DbUserCacheManager] (MSC service thread 1-2) Finished initializing DbUserCacheManager >2014-08-07 14:36:19,244 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (MSC service thread 1-2) Initialization of AsyncTaskManager completed successfully. >2014-08-07 14:36:19,254 INFO [org.ovirt.engine.core.vdsbroker.ResourceManager] (MSC service thread 1-2) Start initializing ResourceManager >2014-08-07 14:36:19,306 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-1) Principal admin::internal synchronized >2014-08-07 14:36:19,387 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (MSC service thread 1-2) Entered VdsManager constructor >2014-08-07 14:36:19,448 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (MSC service thread 1-2) Initialize vdsBroker (10.34.62.208,54,321) >2014-08-07 14:36:19,730 INFO [org.ovirt.engine.core.vdsbroker.ResourceManager] (MSC service thread 1-2) VDS 17f7fe56-5f32-40ef-b098-611b84484e0b was added to the Resource Manager >2014-08-07 14:36:19,804 INFO [org.ovirt.engine.core.vdsbroker.ResourceManager] (MSC service thread 1-2) Finished initializing ResourceManager >2014-08-07 14:36:19,820 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (MSC service thread 1-2) Initialization of OvfDataUpdater completed successfully. >2014-08-07 14:36:19,822 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (MSC service thread 1-2) Start scheduling to enable vds load balancer >2014-08-07 14:36:19,823 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (MSC service thread 1-2) Finished scheduling to enable vds load balancer >2014-08-07 14:36:19,824 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (MSC service thread 1-2) Start HA Reservation check >2014-08-07 14:36:19,830 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (MSC service thread 1-2) Finished HA Reservation check >2014-08-07 14:36:19,841 INFO [org.ovirt.engine.core.bll.network.MacPoolManagerRanges] (org.ovirt.thread.pool-8-thread-1) Start initializing MacPoolManagerRanges >2014-08-07 14:36:19,855 INFO [org.ovirt.engine.core.bll.InitBackendServicesOnStartupBean] (MSC service thread 1-2) Init VM custom properties utilities >2014-08-07 14:36:19,856 INFO [org.ovirt.engine.core.bll.InitBackendServicesOnStartupBean] (MSC service thread 1-2) Init device custom properties utilities >2014-08-07 14:36:19,867 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (MSC service thread 1-2) Initializing Scheduling manager >2014-08-07 14:36:19,874 INFO [org.ovirt.engine.core.bll.network.MacPoolManagerRanges] (org.ovirt.thread.pool-8-thread-1) Finished initializing. Available MACs in pool: 252 >2014-08-07 14:36:19,951 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (MSC service thread 1-2) External scheduler disabled, discovery skipped >2014-08-07 14:36:19,952 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (MSC service thread 1-2) Initialized Scheduling manager >2014-08-07 14:36:19,952 INFO [org.ovirt.engine.core.bll.dwh.DwhHeartBeat] (MSC service thread 1-2) Initializing DWH Heart Beat >2014-08-07 14:36:19,962 INFO [org.ovirt.engine.core.bll.dwh.DwhHeartBeat] (MSC service thread 1-2) DWH Heart Beat initialized >2014-08-07 14:36:20,619 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-2) Running command: LoginUserCommand internal: false. >2014-08-07 14:36:20,625 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-2) Cannot convert the string Host ${VdsName} is not responding. It will stay in Connecting state for a grace period of ${Seconds} seconds and after that an attempt to fence the host will be issued. to AuditLogType, the key does not exist in the AuditLogType declared types >2014-08-07 14:36:20,628 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-2) AuditLogType: UNASSIGNED not exist in string table >2014-08-07 14:36:20,629 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-2) AuditLogType: VDS_AUTO_FENCE_STATUS not exist in string table >2014-08-07 14:36:20,630 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-2) AuditLogType: VDS_AUTO_FENCE_STATUS_FAILED not exist in string table >2014-08-07 14:36:20,630 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-2) AuditLogType: VDS_AUTO_FENCE_FAILED_CALL_FENCE_SPM not exist in string table >2014-08-07 14:36:20,632 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-2) AuditLogType: VDS_HIGH_NETWORK_USE not exist in string table >2014-08-07 14:36:20,632 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-2) AuditLogType: USER_FAILED_REMOVE_VM not exist in string table >2014-08-07 14:36:20,633 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-2) AuditLogType: USER_RUN_UNLOCK_ENTITY_SCRIPT not exist in string table >2014-08-07 14:36:20,634 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-2) AuditLogType: VDS_NETWORK_MTU_DIFFER_FROM_LOGICAL_NETWORK not exist in string table >2014-08-07 14:36:20,638 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-2) AuditLogType: VDS_HOST_IN_CONNECTING_STATE not exist in string table >2014-08-07 14:36:20,639 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-2) AuditLogType: STORAGE_ACTIVATE_ASYNC not exist in string table >2014-08-07 14:36:20,639 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-2) AuditLogType: DWH_STOPPED not exist in string table >2014-08-07 14:36:20,640 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-2) AuditLogType: DWH_STARTED not exist in string table >2014-08-07 14:36:20,646 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-2) AuditLogType: DWH_ERROR not exist in string table >2014-08-07 14:36:20,647 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-2) AuditLogType: USER_REMOVE_AUDIT_LOG not exist in string table >2014-08-07 14:36:20,647 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-2) AuditLogType: USER_REMOVE_AUDIT_LOG_FAILED not exist in string table >2014-08-07 14:36:20,648 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-2) AuditLogType: USER_CLEAR_ALL_DISMISSED_AUDIT_LOG not exist in string table >2014-08-07 14:36:20,648 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-2) AuditLogType: USER_CLEAR_ALL_DISMISSED_AUDIT_LOG_FAILED not exist in string table >2014-08-07 14:36:20,844 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-2) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-08-07 14:36:23,646 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-6) START, GetHardwareInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]), log id: 4f7ba324 >2014-08-07 14:36:23,695 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-6) FINISH, GetHardwareInfoVDSCommand, log id: 4f7ba324 >2014-08-07 14:36:23,974 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-6) Initializing Host: rack >2014-08-07 14:36:24,000 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-6) [326de99b] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS >2014-08-07 14:36:29,939 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-20) [63b0815] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-08-07 14:36:29,960 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-20) [63b0815] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Active to Unknown, reason : null >2014-08-07 14:36:29,967 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-20) [63b0815] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Active to Unknown, reason : null >2014-08-07 14:36:30,040 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-20) [63b0815] Correlation ID: 63b0815, Call Stack: null, Custom Event ID: -1, Message: Invalid status on Data Center Default. Setting status to Non Responsive. >2014-08-07 14:36:30,236 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-20) [63b0815] hostFromVds::selectedVds - rack, spmStatus SPM, storage pool Default >2014-08-07 14:36:30,258 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-20) [63b0815] Initialize Irs proxy from vds: 10.34.62.208 >2014-08-07 14:36:30,293 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-20) [63b0815] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208). >2014-08-07 14:36:30,348 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-13) [63b0815] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 506d2be2 >2014-08-07 14:36:30,462 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-13) [63b0815] -- executeIrsBrokerCommand: Attempting on storage pool 00000002-0002-0002-0002-0000000001d9 >2014-08-07 14:36:30,474 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-13) [63b0815] START, HSMGetAllTasksInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 6f4d7b0b >2014-08-07 14:36:30,488 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-13) [63b0815] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 6f4d7b0b >2014-08-07 14:36:30,490 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-13) [63b0815] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 506d2be2 >2014-08-07 14:36:30,491 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (org.ovirt.thread.pool-8-thread-13) [63b0815] Discovered no tasks on Storage Pool Default >2014-08-07 14:36:45,894 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Loaded file "/usr/share/ovirt-engine/services/ovirt-engine/ovirt-engine.conf". >2014-08-07 14:36:45,895 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) The file "/etc/ovirt-engine/engine.conf" doesn't exist or isn't readable. Will return an empty set of properties. >2014-08-07 14:36:45,895 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Loaded file "/etc/ovirt-engine/engine.conf.d/10-setup-database.conf". >2014-08-07 14:36:45,896 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Loaded file "/etc/ovirt-engine/engine.conf.d/10-setup-jboss.conf". >2014-08-07 14:36:45,897 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Loaded file "/etc/ovirt-engine/engine.conf.d/10-setup-pki.conf". >2014-08-07 14:36:45,897 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Loaded file "/etc/ovirt-engine/engine.conf.d/10-setup-protocols.conf". >2014-08-07 14:36:45,898 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_AJP_ENABLED" is "true". >2014-08-07 14:36:45,898 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_AJP_PORT" is "8702". >2014-08-07 14:36:45,899 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_APPS" is "engine.ear restapi.war legacy_restapi.war". >2014-08-07 14:36:45,899 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_CACHE" is "/var/cache/ovirt-engine". >2014-08-07 14:36:45,904 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_DB_CHECK_INTERVAL" is "1000". >2014-08-07 14:36:45,905 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_DB_CONNECTION_TIMEOUT" is "300000". >2014-08-07 14:36:45,905 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_DB_DATABASE" is "engine". >2014-08-07 14:36:45,905 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_DB_DRIVER" is "org.postgresql.Driver". >2014-08-07 14:36:45,906 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_DB_HOST" is "localhost". >2014-08-07 14:36:45,906 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_DB_MAX_CONNECTIONS" is "100". >2014-08-07 14:36:45,907 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_DB_MIN_CONNECTIONS" is "1". >2014-08-07 14:36:45,907 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_DB_PASSWORD" is "***". >2014-08-07 14:36:45,908 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_DB_PORT" is "5432". >2014-08-07 14:36:45,908 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_DB_SECURED" is "False". >2014-08-07 14:36:45,908 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_DB_SECURED_VALIDATION" is "False". >2014-08-07 14:36:45,909 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_DB_URL" is "jdbc:postgresql://localhost:5432/engine?sslfactory=org.postgresql.ssl.NonValidatingFactory". >2014-08-07 14:36:45,909 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_DB_USER" is "engine". >2014-08-07 14:36:45,910 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_DEBUG_ADDRESS" is "". >2014-08-07 14:36:45,910 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_DOC" is "/usr/share/doc/ovirt-engine". >2014-08-07 14:36:45,911 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_ETC" is "/etc/ovirt-engine". >2014-08-07 14:36:45,911 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_EXTENSION_PATH" is "/usr/share/ovirt-engine/extensions.d:/etc/ovirt-engine/extensions.d". >2014-08-07 14:36:45,912 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_FQDN" is "pb-rh35.rhev.lab.eng.brq.redhat.com". >2014-08-07 14:36:45,912 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_GROUP" is "ovirt". >2014-08-07 14:36:45,913 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_HEAP_MAX" is "1g". >2014-08-07 14:36:45,930 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_HEAP_MIN" is "1g". >2014-08-07 14:36:45,931 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_HTTPS_ENABLED" is "false". >2014-08-07 14:36:45,931 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_HTTPS_PORT" is "None". >2014-08-07 14:36:45,932 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_HTTPS_PROTOCOLS" is "SSLv3,TLSv1,TLSv1.1,TLSv1.2". >2014-08-07 14:36:45,932 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_HTTP_ENABLED" is "false". >2014-08-07 14:36:45,932 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_HTTP_PORT" is "None". >2014-08-07 14:36:45,933 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_JAVA_MODULEPATH" is "/usr/share/ovirt-engine/modules". >2014-08-07 14:36:45,933 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_JVM_ARGS" is " -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath="/var/log/ovirt-engine/dump"". >2014-08-07 14:36:45,934 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_LOG" is "/var/log/ovirt-engine". >2014-08-07 14:36:45,934 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_LOG_TO_CONSOLE" is "false". >2014-08-07 14:36:45,935 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_MANUAL" is "/usr/share/ovirt-engine/manual". >2014-08-07 14:36:45,935 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_PERM_MAX" is "256m". >2014-08-07 14:36:45,936 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_PERM_MIN" is "256m". >2014-08-07 14:36:45,936 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_PKI" is "/etc/pki/ovirt-engine". >2014-08-07 14:36:45,937 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_PKI_CA" is "/etc/pki/ovirt-engine/ca.pem". >2014-08-07 14:36:45,937 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_PKI_ENGINE_CERT" is "/etc/pki/ovirt-engine/certs/engine.cer". >2014-08-07 14:36:45,938 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_PKI_ENGINE_STORE" is "/etc/pki/ovirt-engine/keys/engine.p12". >2014-08-07 14:36:45,938 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_PKI_ENGINE_STORE_ALIAS" is "1". >2014-08-07 14:36:45,939 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_PKI_ENGINE_STORE_PASSWORD" is "***". >2014-08-07 14:36:45,939 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_PKI_TRUST_STORE" is "/etc/pki/ovirt-engine/.truststore". >2014-08-07 14:36:45,940 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_PKI_TRUST_STORE_PASSWORD" is "***". >2014-08-07 14:36:45,942 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_PROPERTIES" is " jsse.enableSNIExtension=false "java.security.krb5.conf=/etc/ovirt-engine/krb5.conf"". >2014-08-07 14:36:45,943 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_PROXY_ENABLED" is "true". >2014-08-07 14:36:45,943 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_PROXY_HTTPS_PORT" is "443". >2014-08-07 14:36:45,944 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_PROXY_HTTP_PORT" is "80". >2014-08-07 14:36:45,944 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_REPORTS_UI" is "/var/lib/ovirt-engine/reports.xml". >2014-08-07 14:36:45,944 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_STOP_INTERVAL" is "1". >2014-08-07 14:36:45,945 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_STOP_TIME" is "10". >2014-08-07 14:36:45,945 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_TMP" is "/var/tmp/ovirt-engine". >2014-08-07 14:36:45,946 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_UP_MARK" is "/var/lib/ovirt-engine/engine.up". >2014-08-07 14:36:45,946 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_URI" is "/ovirt-engine". >2014-08-07 14:36:45,947 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_USER" is "ovirt". >2014-08-07 14:36:45,947 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_USR" is "/usr/share/ovirt-engine". >2014-08-07 14:36:45,949 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_VAR" is "/var/lib/ovirt-engine". >2014-08-07 14:36:45,949 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_VERBOSE_GC" is "false". >2014-08-07 14:36:45,953 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "JBOSS_HOME" is "/usr/share/ovirt-engine-jboss-as". >2014-08-07 14:36:45,953 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "JBOSS_RUNTIME" is "/var/lib/ovirt-engine/jboss_runtime". >2014-08-07 14:36:45,954 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "SENSITIVE_KEYS" is ",ENGINE_DB_PASSWORD,ENGINE_PKI_TRUST_STORE_PASSWORD,ENGINE_PKI_ENGINE_STORE_PASSWORD". >2014-08-07 14:36:46,575 INFO [org.ovirt.engine.core.bll.Backend] (MSC service thread 1-1) Start initializing Backend >2014-08-07 14:36:47,147 INFO [org.ovirt.engine.core.utils.osinfo.OsInfoPreferencesLoader] (MSC service thread 1-1) Loaded file /etc/ovirt-engine/osinfo.conf.d/00-defaults.properties >2014-08-07 14:36:47,273 INFO [org.ovirt.engine.core.bll.Backend] (MSC service thread 1-1) Running ovirt-engine 3.5.0-0.0.master.20140804172041.git23b558e.el6 >2014-08-07 14:36:47,274 INFO [org.ovirt.engine.core.bll.CpuFlagsManagerHandler] (MSC service thread 1-1) Start initializing dictionaries >2014-08-07 14:36:47,278 INFO [org.ovirt.engine.core.bll.CpuFlagsManagerHandler] (MSC service thread 1-1) Finished initializing dictionaries >2014-08-07 14:36:47,279 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (MSC service thread 1-1) Start initializing AuditLogCleanupManager >2014-08-07 14:36:47,280 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (MSC service thread 1-1) Setting audit cleanup manager to run at: 35 35 3 * * ? >2014-08-07 14:36:47,312 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (MSC service thread 1-1) Finished initializing AuditLogCleanupManager >2014-08-07 14:36:47,313 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (MSC service thread 1-1) Start initializing CommandEntityCleanupManager >2014-08-07 14:36:47,314 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (MSC service thread 1-1) Setting command entity cleanup manager to run at: 35 35 3 * * ? >2014-08-07 14:36:47,321 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (MSC service thread 1-1) Finished initializing CommandEntityCleanupManager >2014-08-07 14:36:47,324 INFO [org.ovirt.engine.core.bll.TagsDirector] (MSC service thread 1-1) Start initializing TagsDirector >2014-08-07 14:36:47,342 INFO [org.ovirt.engine.core.bll.TagsDirector] (MSC service thread 1-1) Tag root added to tree >2014-08-07 14:36:47,353 INFO [org.ovirt.engine.core.bll.TagsDirector] (MSC service thread 1-1) Finished initializing TagsDirector >2014-08-07 14:36:47,354 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (MSC service thread 1-1) Start initializing IsoDomainListSyncronizer >2014-08-07 14:36:47,364 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (MSC service thread 1-1) Finished initializing IsoDomainListSyncronizer >2014-08-07 14:36:47,456 INFO [org.ovirt.engine.core.bll.Backend] (MSC service thread 1-1) Completed initializing handlers >2014-08-07 14:36:47,493 INFO [org.ovirt.engine.core.utils.ErrorTranslatorImpl] (MSC service thread 1-1) Start initializing ErrorTranslatorImpl >2014-08-07 14:36:47,505 WARN [org.ovirt.engine.core.utils.ErrorTranslatorImpl] (MSC service thread 1-1) Code MAC_ADDRESS_IS_IN_USE appears more than once in string table. >2014-08-07 14:36:47,507 INFO [org.ovirt.engine.core.utils.ErrorTranslatorImpl] (MSC service thread 1-1) Finished initializing ErrorTranslatorImpl >2014-08-07 14:36:47,507 INFO [org.ovirt.engine.core.utils.ErrorTranslatorImpl] (MSC service thread 1-1) Start initializing ErrorTranslatorImpl >2014-08-07 14:36:47,508 INFO [org.ovirt.engine.core.utils.ErrorTranslatorImpl] (MSC service thread 1-1) Finished initializing ErrorTranslatorImpl >2014-08-07 14:36:47,509 INFO [org.ovirt.engine.core.bll.Backend] (MSC service thread 1-1) Mark incomplete jobs as UNKNOWN >2014-08-07 14:36:47,544 INFO [org.ovirt.engine.core.bll.job.JobRepositoryCleanupManager] (MSC service thread 1-1) Start initializing JobRepositoryCleanupManager >2014-08-07 14:36:47,545 INFO [org.ovirt.engine.core.bll.job.JobRepositoryCleanupManager] (MSC service thread 1-1) Finished initializing JobRepositoryCleanupManager >2014-08-07 14:36:47,546 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (MSC service thread 1-1) Start initializing AutoRecoveryManager >2014-08-07 14:36:47,547 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (MSC service thread 1-1) Finished initializing AutoRecoveryManager >2014-08-07 14:36:47,550 INFO [org.ovirt.engine.core.dal.job.ExecutionMessageDirector] (MSC service thread 1-1) Start initializing ExecutionMessageDirector >2014-08-07 14:36:47,553 INFO [org.ovirt.engine.core.dal.job.ExecutionMessageDirector] (MSC service thread 1-1) Finished initializing ExecutionMessageDirector >2014-08-07 14:36:47,601 INFO [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (MSC service thread 1-1) Start of enabled extensions list >2014-08-07 14:36:47,602 INFO [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (MSC service thread 1-1) Instance name: 'builtin-authn-internal', Extension name: 'Internal Authn (Built-in)', Version: 'N/A', Notes: '', License: 'ASL 2.0', Home: 'http://www.ovirt.org', Author 'The oVirt Project', Build interface Version: '0', File: 'N/A', Initialized: 'true' >2014-08-07 14:36:47,602 INFO [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (MSC service thread 1-1) Instance name: 'internal', Extension name: 'Internal Authz (Built-in)', Version: 'N/A', Notes: '', License: 'ASL 2.0', Home: 'http://www.ovirt.org', Author 'The oVirt Project', Build interface Version: '0', File: 'N/A', Initialized: 'true' >2014-08-07 14:36:47,604 INFO [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (MSC service thread 1-1) End of enabled extensions list >2014-08-07 14:36:47,723 INFO [org.ovirt.engine.core.bll.aaa.DbUserCacheManager] (MSC service thread 1-3) Start initializing DbUserCacheManager >2014-08-07 14:36:47,733 INFO [org.ovirt.engine.core.bll.aaa.DbUserCacheManager] (MSC service thread 1-3) Finished initializing DbUserCacheManager >2014-08-07 14:36:47,747 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (MSC service thread 1-3) Initialization of AsyncTaskManager completed successfully. >2014-08-07 14:36:47,749 INFO [org.ovirt.engine.core.vdsbroker.ResourceManager] (MSC service thread 1-3) Start initializing ResourceManager >2014-08-07 14:36:47,779 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-1) Principal admin::internal synchronized >2014-08-07 14:36:47,842 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (MSC service thread 1-3) Entered VdsManager constructor >2014-08-07 14:36:47,908 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (MSC service thread 1-3) Initialize vdsBroker (10.34.62.208,54,321) >2014-08-07 14:36:47,925 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-6) Running command: LoginUserCommand internal: false. >2014-08-07 14:36:47,939 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) Cannot convert the string Host ${VdsName} is not responding. It will stay in Connecting state for a grace period of ${Seconds} seconds and after that an attempt to fence the host will be issued. to AuditLogType, the key does not exist in the AuditLogType declared types >2014-08-07 14:36:47,947 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) AuditLogType: UNASSIGNED not exist in string table >2014-08-07 14:36:47,947 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) AuditLogType: VDS_AUTO_FENCE_STATUS not exist in string table >2014-08-07 14:36:47,948 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) AuditLogType: VDS_AUTO_FENCE_STATUS_FAILED not exist in string table >2014-08-07 14:36:47,948 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) AuditLogType: VDS_AUTO_FENCE_FAILED_CALL_FENCE_SPM not exist in string table >2014-08-07 14:36:47,958 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) AuditLogType: VDS_HIGH_NETWORK_USE not exist in string table >2014-08-07 14:36:47,959 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) AuditLogType: USER_FAILED_REMOVE_VM not exist in string table >2014-08-07 14:36:47,959 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) AuditLogType: USER_RUN_UNLOCK_ENTITY_SCRIPT not exist in string table >2014-08-07 14:36:47,960 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) AuditLogType: VDS_NETWORK_MTU_DIFFER_FROM_LOGICAL_NETWORK not exist in string table >2014-08-07 14:36:47,961 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) AuditLogType: VDS_HOST_IN_CONNECTING_STATE not exist in string table >2014-08-07 14:36:47,968 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) AuditLogType: STORAGE_ACTIVATE_ASYNC not exist in string table >2014-08-07 14:36:47,969 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) AuditLogType: DWH_STOPPED not exist in string table >2014-08-07 14:36:47,970 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) AuditLogType: DWH_STARTED not exist in string table >2014-08-07 14:36:47,970 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) AuditLogType: DWH_ERROR not exist in string table >2014-08-07 14:36:47,971 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) AuditLogType: USER_REMOVE_AUDIT_LOG not exist in string table >2014-08-07 14:36:47,971 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) AuditLogType: USER_REMOVE_AUDIT_LOG_FAILED not exist in string table >2014-08-07 14:36:47,972 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) AuditLogType: USER_CLEAR_ALL_DISMISSED_AUDIT_LOG not exist in string table >2014-08-07 14:36:47,980 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) AuditLogType: USER_CLEAR_ALL_DISMISSED_AUDIT_LOG_FAILED not exist in string table >2014-08-07 14:36:48,244 INFO [org.ovirt.engine.core.vdsbroker.ResourceManager] (MSC service thread 1-3) VDS 17f7fe56-5f32-40ef-b098-611b84484e0b was added to the Resource Manager >2014-08-07 14:36:48,287 INFO [org.ovirt.engine.core.vdsbroker.ResourceManager] (MSC service thread 1-3) Finished initializing ResourceManager >2014-08-07 14:36:48,295 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (MSC service thread 1-3) Initialization of OvfDataUpdater completed successfully. >2014-08-07 14:36:48,295 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (MSC service thread 1-3) Start scheduling to enable vds load balancer >2014-08-07 14:36:48,297 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (MSC service thread 1-3) Finished scheduling to enable vds load balancer >2014-08-07 14:36:48,297 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (MSC service thread 1-3) Start HA Reservation check >2014-08-07 14:36:48,298 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (MSC service thread 1-3) Finished HA Reservation check >2014-08-07 14:36:48,307 INFO [org.ovirt.engine.core.bll.network.MacPoolManagerRanges] (org.ovirt.thread.pool-8-thread-2) Start initializing MacPoolManagerRanges >2014-08-07 14:36:48,316 INFO [org.ovirt.engine.core.bll.InitBackendServicesOnStartupBean] (MSC service thread 1-3) Init VM custom properties utilities >2014-08-07 14:36:48,318 INFO [org.ovirt.engine.core.bll.InitBackendServicesOnStartupBean] (MSC service thread 1-3) Init device custom properties utilities >2014-08-07 14:36:48,330 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (MSC service thread 1-3) Initializing Scheduling manager >2014-08-07 14:36:48,343 INFO [org.ovirt.engine.core.bll.network.MacPoolManagerRanges] (org.ovirt.thread.pool-8-thread-2) Finished initializing. Available MACs in pool: 252 >2014-08-07 14:36:48,392 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (MSC service thread 1-3) External scheduler disabled, discovery skipped >2014-08-07 14:36:48,393 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (MSC service thread 1-3) Initialized Scheduling manager >2014-08-07 14:36:48,394 INFO [org.ovirt.engine.core.bll.dwh.DwhHeartBeat] (MSC service thread 1-3) Initializing DWH Heart Beat >2014-08-07 14:36:48,401 INFO [org.ovirt.engine.core.bll.dwh.DwhHeartBeat] (MSC service thread 1-3) DWH Heart Beat initialized >2014-08-07 14:36:48,460 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-08-07 14:36:52,076 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-6) START, GetHardwareInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]), log id: 32a6111c >2014-08-07 14:36:52,133 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-6) FINISH, GetHardwareInfoVDSCommand, log id: 32a6111c >2014-08-07 14:36:52,434 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-6) Initializing Host: rack >2014-08-07 14:36:52,453 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-6) [22a9866e] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS >2014-08-07 14:36:58,429 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-20) [6e21a806] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-08-07 14:36:58,444 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-20) [6e21a806] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Active to Unknown, reason : null >2014-08-07 14:36:58,453 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-20) [6e21a806] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Active to Unknown, reason : null >2014-08-07 14:36:58,569 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-20) [6e21a806] Correlation ID: 6e21a806, Call Stack: null, Custom Event ID: -1, Message: Invalid status on Data Center Default. Setting status to Non Responsive. >2014-08-07 14:36:58,746 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-20) [6e21a806] hostFromVds::selectedVds - rack, spmStatus SPM, storage pool Default >2014-08-07 14:36:58,767 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-20) [6e21a806] Initialize Irs proxy from vds: 10.34.62.208 >2014-08-07 14:36:58,791 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-20) [6e21a806] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208). >2014-08-07 14:36:58,919 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-15) [6e21a806] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 6e3d4086 >2014-08-07 14:36:58,921 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-15) [6e21a806] -- executeIrsBrokerCommand: Attempting on storage pool 00000002-0002-0002-0002-0000000001d9 >2014-08-07 14:36:58,945 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-15) [6e21a806] START, HSMGetAllTasksInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 657ce6b0 >2014-08-07 14:36:58,956 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-15) [6e21a806] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 657ce6b0 >2014-08-07 14:36:58,958 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-15) [6e21a806] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 6e3d4086 >2014-08-07 14:36:58,959 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (org.ovirt.thread.pool-8-thread-15) [6e21a806] Discovered no tasks on Storage Pool Default >2014-08-07 14:37:22,290 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Loaded file "/usr/share/ovirt-engine/services/ovirt-engine/ovirt-engine.conf". >2014-08-07 14:37:22,291 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) The file "/etc/ovirt-engine/engine.conf" doesn't exist or isn't readable. Will return an empty set of properties. >2014-08-07 14:37:22,292 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Loaded file "/etc/ovirt-engine/engine.conf.d/10-setup-database.conf". >2014-08-07 14:37:22,292 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Loaded file "/etc/ovirt-engine/engine.conf.d/10-setup-jboss.conf". >2014-08-07 14:37:22,293 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Loaded file "/etc/ovirt-engine/engine.conf.d/10-setup-pki.conf". >2014-08-07 14:37:22,300 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Loaded file "/etc/ovirt-engine/engine.conf.d/10-setup-protocols.conf". >2014-08-07 14:37:22,302 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_AJP_ENABLED" is "true". >2014-08-07 14:37:22,303 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_AJP_PORT" is "8702". >2014-08-07 14:37:22,303 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_APPS" is "engine.ear restapi.war legacy_restapi.war". >2014-08-07 14:37:22,304 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_CACHE" is "/var/cache/ovirt-engine". >2014-08-07 14:37:22,305 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_DB_CHECK_INTERVAL" is "1000". >2014-08-07 14:37:22,305 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_DB_CONNECTION_TIMEOUT" is "300000". >2014-08-07 14:37:22,306 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_DB_DATABASE" is "engine". >2014-08-07 14:37:22,306 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_DB_DRIVER" is "org.postgresql.Driver". >2014-08-07 14:37:22,307 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_DB_HOST" is "localhost". >2014-08-07 14:37:22,313 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_DB_MAX_CONNECTIONS" is "100". >2014-08-07 14:37:22,314 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_DB_MIN_CONNECTIONS" is "1". >2014-08-07 14:37:22,315 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_DB_PASSWORD" is "***". >2014-08-07 14:37:22,315 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_DB_PORT" is "5432". >2014-08-07 14:37:22,315 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_DB_SECURED" is "False". >2014-08-07 14:37:22,316 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_DB_SECURED_VALIDATION" is "False". >2014-08-07 14:37:22,316 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_DB_URL" is "jdbc:postgresql://localhost:5432/engine?sslfactory=org.postgresql.ssl.NonValidatingFactory". >2014-08-07 14:37:22,317 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_DB_USER" is "engine". >2014-08-07 14:37:22,317 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_DEBUG_ADDRESS" is "". >2014-08-07 14:37:22,318 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_DOC" is "/usr/share/doc/ovirt-engine". >2014-08-07 14:37:22,318 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_ETC" is "/etc/ovirt-engine". >2014-08-07 14:37:22,319 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_EXTENSION_PATH" is "/usr/share/ovirt-engine/extensions.d:/etc/ovirt-engine/extensions.d". >2014-08-07 14:37:22,323 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_FQDN" is "pb-rh35.rhev.lab.eng.brq.redhat.com". >2014-08-07 14:37:22,323 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_GROUP" is "ovirt". >2014-08-07 14:37:22,324 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_HEAP_MAX" is "1g". >2014-08-07 14:37:22,324 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_HEAP_MIN" is "1g". >2014-08-07 14:37:22,325 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_HTTPS_ENABLED" is "false". >2014-08-07 14:37:22,325 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_HTTPS_PORT" is "None". >2014-08-07 14:37:22,326 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_HTTPS_PROTOCOLS" is "SSLv3,TLSv1,TLSv1.1,TLSv1.2". >2014-08-07 14:37:22,327 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_HTTP_ENABLED" is "false". >2014-08-07 14:37:22,327 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_HTTP_PORT" is "None". >2014-08-07 14:37:22,334 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_JAVA_MODULEPATH" is "/usr/share/ovirt-engine/modules". >2014-08-07 14:37:22,336 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_JVM_ARGS" is " -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath="/var/log/ovirt-engine/dump"". >2014-08-07 14:37:22,337 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_LOG" is "/var/log/ovirt-engine". >2014-08-07 14:37:22,337 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_LOG_TO_CONSOLE" is "false". >2014-08-07 14:37:22,337 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_MANUAL" is "/usr/share/ovirt-engine/manual". >2014-08-07 14:37:22,338 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_PERM_MAX" is "256m". >2014-08-07 14:37:22,338 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_PERM_MIN" is "256m". >2014-08-07 14:37:22,339 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_PKI" is "/etc/pki/ovirt-engine". >2014-08-07 14:37:22,339 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_PKI_CA" is "/etc/pki/ovirt-engine/ca.pem". >2014-08-07 14:37:22,340 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_PKI_ENGINE_CERT" is "/etc/pki/ovirt-engine/certs/engine.cer". >2014-08-07 14:37:22,340 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_PKI_ENGINE_STORE" is "/etc/pki/ovirt-engine/keys/engine.p12". >2014-08-07 14:37:22,341 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_PKI_ENGINE_STORE_ALIAS" is "1". >2014-08-07 14:37:22,348 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_PKI_ENGINE_STORE_PASSWORD" is "***". >2014-08-07 14:37:22,349 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_PKI_TRUST_STORE" is "/etc/pki/ovirt-engine/.truststore". >2014-08-07 14:37:22,350 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_PKI_TRUST_STORE_PASSWORD" is "***". >2014-08-07 14:37:22,350 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_PROPERTIES" is " jsse.enableSNIExtension=false "java.security.krb5.conf=/etc/ovirt-engine/krb5.conf"". >2014-08-07 14:37:22,351 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_PROXY_ENABLED" is "true". >2014-08-07 14:37:22,351 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_PROXY_HTTPS_PORT" is "443". >2014-08-07 14:37:22,352 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_PROXY_HTTP_PORT" is "80". >2014-08-07 14:37:22,352 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_REPORTS_UI" is "/var/lib/ovirt-engine/reports.xml". >2014-08-07 14:37:22,353 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_STOP_INTERVAL" is "1". >2014-08-07 14:37:22,353 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_STOP_TIME" is "10". >2014-08-07 14:37:22,353 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_TMP" is "/var/tmp/ovirt-engine". >2014-08-07 14:37:22,355 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_UP_MARK" is "/var/lib/ovirt-engine/engine.up". >2014-08-07 14:37:22,359 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_URI" is "/ovirt-engine". >2014-08-07 14:37:22,359 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_USER" is "ovirt". >2014-08-07 14:37:22,360 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_USR" is "/usr/share/ovirt-engine". >2014-08-07 14:37:22,360 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_VAR" is "/var/lib/ovirt-engine". >2014-08-07 14:37:22,366 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "ENGINE_VERBOSE_GC" is "false". >2014-08-07 14:37:22,367 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "JBOSS_HOME" is "/usr/share/ovirt-engine-jboss-as". >2014-08-07 14:37:22,367 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "JBOSS_RUNTIME" is "/var/lib/ovirt-engine/jboss_runtime". >2014-08-07 14:37:22,368 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-2) Value of property "SENSITIVE_KEYS" is ",ENGINE_DB_PASSWORD,ENGINE_PKI_TRUST_STORE_PASSWORD,ENGINE_PKI_ENGINE_STORE_PASSWORD". >2014-08-07 14:37:23,197 INFO [org.ovirt.engine.core.bll.Backend] (MSC service thread 1-2) Start initializing Backend >2014-08-07 14:37:23,858 INFO [org.ovirt.engine.core.utils.osinfo.OsInfoPreferencesLoader] (MSC service thread 1-2) Loaded file /etc/ovirt-engine/osinfo.conf.d/00-defaults.properties >2014-08-07 14:37:23,927 ERROR [org.ovirt.engine.core.aaa.filters.BasicAuthenticationFilter] (ajp--127.0.0.1-8702-1) Error in obtaining profile null >2014-08-07 14:37:24,031 INFO [org.ovirt.engine.core.bll.Backend] (MSC service thread 1-2) Running ovirt-engine 3.5.0-0.0.master.20140804172041.git23b558e.el6 >2014-08-07 14:37:24,032 INFO [org.ovirt.engine.core.bll.CpuFlagsManagerHandler] (MSC service thread 1-2) Start initializing dictionaries >2014-08-07 14:37:24,037 INFO [org.ovirt.engine.core.bll.CpuFlagsManagerHandler] (MSC service thread 1-2) Finished initializing dictionaries >2014-08-07 14:37:24,038 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (MSC service thread 1-2) Start initializing AuditLogCleanupManager >2014-08-07 14:37:24,040 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (MSC service thread 1-2) Setting audit cleanup manager to run at: 35 35 3 * * ? >2014-08-07 14:37:24,063 ERROR [org.ovirt.engine.core.aaa.filters.BasicAuthenticationFilter] (ajp--127.0.0.1-8702-2) Error in obtaining profile null >2014-08-07 14:37:24,096 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (MSC service thread 1-2) Finished initializing AuditLogCleanupManager >2014-08-07 14:37:24,097 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (MSC service thread 1-2) Start initializing CommandEntityCleanupManager >2014-08-07 14:37:24,098 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (MSC service thread 1-2) Setting command entity cleanup manager to run at: 35 35 3 * * ? >2014-08-07 14:37:24,108 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (MSC service thread 1-2) Finished initializing CommandEntityCleanupManager >2014-08-07 14:37:24,110 INFO [org.ovirt.engine.core.bll.TagsDirector] (MSC service thread 1-2) Start initializing TagsDirector >2014-08-07 14:37:24,128 INFO [org.ovirt.engine.core.bll.TagsDirector] (MSC service thread 1-2) Tag root added to tree >2014-08-07 14:37:24,146 INFO [org.ovirt.engine.core.bll.TagsDirector] (MSC service thread 1-2) Finished initializing TagsDirector >2014-08-07 14:37:24,147 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (MSC service thread 1-2) Start initializing IsoDomainListSyncronizer >2014-08-07 14:37:24,159 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (MSC service thread 1-2) Finished initializing IsoDomainListSyncronizer >2014-08-07 14:37:24,195 ERROR [org.ovirt.engine.core.aaa.filters.BasicAuthenticationFilter] (ajp--127.0.0.1-8702-3) Error in obtaining profile null >2014-08-07 14:37:24,293 INFO [org.ovirt.engine.core.bll.Backend] (MSC service thread 1-2) Completed initializing handlers >2014-08-07 14:37:24,325 ERROR [org.ovirt.engine.core.aaa.filters.BasicAuthenticationFilter] (ajp--127.0.0.1-8702-4) Error in obtaining profile null >2014-08-07 14:37:24,351 INFO [org.ovirt.engine.core.utils.ErrorTranslatorImpl] (MSC service thread 1-2) Start initializing ErrorTranslatorImpl >2014-08-07 14:37:24,375 WARN [org.ovirt.engine.core.utils.ErrorTranslatorImpl] (MSC service thread 1-2) Code MAC_ADDRESS_IS_IN_USE appears more than once in string table. >2014-08-07 14:37:24,376 INFO [org.ovirt.engine.core.utils.ErrorTranslatorImpl] (MSC service thread 1-2) Finished initializing ErrorTranslatorImpl >2014-08-07 14:37:24,377 INFO [org.ovirt.engine.core.utils.ErrorTranslatorImpl] (MSC service thread 1-2) Start initializing ErrorTranslatorImpl >2014-08-07 14:37:24,378 INFO [org.ovirt.engine.core.utils.ErrorTranslatorImpl] (MSC service thread 1-2) Finished initializing ErrorTranslatorImpl >2014-08-07 14:37:24,379 INFO [org.ovirt.engine.core.bll.Backend] (MSC service thread 1-2) Mark incomplete jobs as UNKNOWN >2014-08-07 14:37:24,420 INFO [org.ovirt.engine.core.bll.job.JobRepositoryCleanupManager] (MSC service thread 1-2) Start initializing JobRepositoryCleanupManager >2014-08-07 14:37:24,422 INFO [org.ovirt.engine.core.bll.job.JobRepositoryCleanupManager] (MSC service thread 1-2) Finished initializing JobRepositoryCleanupManager >2014-08-07 14:37:24,423 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (MSC service thread 1-2) Start initializing AutoRecoveryManager >2014-08-07 14:37:24,426 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (MSC service thread 1-2) Finished initializing AutoRecoveryManager >2014-08-07 14:37:24,451 ERROR [org.ovirt.engine.core.aaa.filters.BasicAuthenticationFilter] (ajp--127.0.0.1-8702-5) Error in obtaining profile null >2014-08-07 14:37:24,450 INFO [org.ovirt.engine.core.dal.job.ExecutionMessageDirector] (MSC service thread 1-2) Start initializing ExecutionMessageDirector >2014-08-07 14:37:24,455 INFO [org.ovirt.engine.core.dal.job.ExecutionMessageDirector] (MSC service thread 1-2) Finished initializing ExecutionMessageDirector >2014-08-07 14:37:24,523 INFO [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (MSC service thread 1-2) Start of enabled extensions list >2014-08-07 14:37:24,524 INFO [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (MSC service thread 1-2) Instance name: 'builtin-authn-internal', Extension name: 'Internal Authn (Built-in)', Version: 'N/A', Notes: '', License: 'ASL 2.0', Home: 'http://www.ovirt.org', Author 'The oVirt Project', Build interface Version: '0', File: 'N/A', Initialized: 'true' >2014-08-07 14:37:24,526 INFO [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (MSC service thread 1-2) Instance name: 'internal', Extension name: 'Internal Authz (Built-in)', Version: 'N/A', Notes: '', License: 'ASL 2.0', Home: 'http://www.ovirt.org', Author 'The oVirt Project', Build interface Version: '0', File: 'N/A', Initialized: 'true' >2014-08-07 14:37:24,527 INFO [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (MSC service thread 1-2) End of enabled extensions list >2014-08-07 14:37:24,631 INFO [org.ovirt.engine.core.bll.aaa.DbUserCacheManager] (MSC service thread 1-1) Start initializing DbUserCacheManager >2014-08-07 14:37:24,645 INFO [org.ovirt.engine.core.bll.aaa.DbUserCacheManager] (MSC service thread 1-1) Finished initializing DbUserCacheManager >2014-08-07 14:37:24,680 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (MSC service thread 1-1) Initialization of AsyncTaskManager completed successfully. >2014-08-07 14:37:24,702 INFO [org.ovirt.engine.core.vdsbroker.ResourceManager] (MSC service thread 1-1) Start initializing ResourceManager >2014-08-07 14:37:24,716 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-1) Principal admin::internal synchronized >2014-08-07 14:37:24,819 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (MSC service thread 1-1) Entered VdsManager constructor >2014-08-07 14:37:24,888 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (MSC service thread 1-1) Initialize vdsBroker (10.34.62.208,54,321) >2014-08-07 14:37:24,896 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-6) Running command: LoginUserCommand internal: false. >2014-08-07 14:37:24,911 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) Cannot convert the string Host ${VdsName} is not responding. It will stay in Connecting state for a grace period of ${Seconds} seconds and after that an attempt to fence the host will be issued. to AuditLogType, the key does not exist in the AuditLogType declared types >2014-08-07 14:37:24,914 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) AuditLogType: UNASSIGNED not exist in string table >2014-08-07 14:37:24,915 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) AuditLogType: VDS_AUTO_FENCE_STATUS not exist in string table >2014-08-07 14:37:24,916 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) AuditLogType: VDS_AUTO_FENCE_STATUS_FAILED not exist in string table >2014-08-07 14:37:24,916 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) AuditLogType: VDS_AUTO_FENCE_FAILED_CALL_FENCE_SPM not exist in string table >2014-08-07 14:37:24,923 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) AuditLogType: VDS_HIGH_NETWORK_USE not exist in string table >2014-08-07 14:37:24,924 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) AuditLogType: USER_FAILED_REMOVE_VM not exist in string table >2014-08-07 14:37:24,924 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) AuditLogType: USER_RUN_UNLOCK_ENTITY_SCRIPT not exist in string table >2014-08-07 14:37:24,925 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) AuditLogType: VDS_NETWORK_MTU_DIFFER_FROM_LOGICAL_NETWORK not exist in string table >2014-08-07 14:37:24,926 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) AuditLogType: VDS_HOST_IN_CONNECTING_STATE not exist in string table >2014-08-07 14:37:24,926 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) AuditLogType: STORAGE_ACTIVATE_ASYNC not exist in string table >2014-08-07 14:37:24,927 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) AuditLogType: DWH_STOPPED not exist in string table >2014-08-07 14:37:24,928 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) AuditLogType: DWH_STARTED not exist in string table >2014-08-07 14:37:24,928 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) AuditLogType: DWH_ERROR not exist in string table >2014-08-07 14:37:24,937 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) AuditLogType: USER_REMOVE_AUDIT_LOG not exist in string table >2014-08-07 14:37:24,937 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) AuditLogType: USER_REMOVE_AUDIT_LOG_FAILED not exist in string table >2014-08-07 14:37:24,938 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) AuditLogType: USER_CLEAR_ALL_DISMISSED_AUDIT_LOG not exist in string table >2014-08-07 14:37:24,938 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) AuditLogType: USER_CLEAR_ALL_DISMISSED_AUDIT_LOG_FAILED not exist in string table >2014-08-07 14:37:25,043 INFO [org.ovirt.engine.core.vdsbroker.ResourceManager] (MSC service thread 1-1) VDS 17f7fe56-5f32-40ef-b098-611b84484e0b was added to the Resource Manager >2014-08-07 14:37:25,099 INFO [org.ovirt.engine.core.vdsbroker.ResourceManager] (MSC service thread 1-1) Finished initializing ResourceManager >2014-08-07 14:37:25,116 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (MSC service thread 1-1) Initialization of OvfDataUpdater completed successfully. >2014-08-07 14:37:25,117 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (MSC service thread 1-1) Start scheduling to enable vds load balancer >2014-08-07 14:37:25,119 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (MSC service thread 1-1) Finished scheduling to enable vds load balancer >2014-08-07 14:37:25,119 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (MSC service thread 1-1) Start HA Reservation check >2014-08-07 14:37:25,120 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (MSC service thread 1-1) Finished HA Reservation check >2014-08-07 14:37:25,129 INFO [org.ovirt.engine.core.bll.network.MacPoolManagerRanges] (org.ovirt.thread.pool-8-thread-2) Start initializing MacPoolManagerRanges >2014-08-07 14:37:25,157 INFO [org.ovirt.engine.core.bll.InitBackendServicesOnStartupBean] (MSC service thread 1-1) Init VM custom properties utilities >2014-08-07 14:37:25,160 INFO [org.ovirt.engine.core.bll.InitBackendServicesOnStartupBean] (MSC service thread 1-1) Init device custom properties utilities >2014-08-07 14:37:25,173 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (MSC service thread 1-1) Initializing Scheduling manager >2014-08-07 14:37:25,175 INFO [org.ovirt.engine.core.bll.network.MacPoolManagerRanges] (org.ovirt.thread.pool-8-thread-2) Finished initializing. Available MACs in pool: 252 >2014-08-07 14:37:25,230 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (MSC service thread 1-1) External scheduler disabled, discovery skipped >2014-08-07 14:37:25,231 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (MSC service thread 1-1) Initialized Scheduling manager >2014-08-07 14:37:25,232 INFO [org.ovirt.engine.core.bll.dwh.DwhHeartBeat] (MSC service thread 1-1) Initializing DWH Heart Beat >2014-08-07 14:37:25,238 INFO [org.ovirt.engine.core.bll.dwh.DwhHeartBeat] (MSC service thread 1-1) DWH Heart Beat initialized >2014-08-07 14:37:25,414 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-08-07 14:37:28,846 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-6) START, GetHardwareInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]), log id: b98d041 >2014-08-07 14:37:28,894 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-6) FINISH, GetHardwareInfoVDSCommand, log id: b98d041 >2014-08-07 14:37:29,180 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-6) Initializing Host: rack >2014-08-07 14:37:29,203 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-6) [44bd54c7] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS >2014-08-07 14:37:35,166 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-20) [32f66d2c] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-08-07 14:37:35,176 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-20) [32f66d2c] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Active to Unknown, reason : null >2014-08-07 14:37:35,187 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-20) [32f66d2c] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Active to Unknown, reason : null >2014-08-07 14:37:35,230 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-20) [32f66d2c] Correlation ID: 32f66d2c, Call Stack: null, Custom Event ID: -1, Message: Invalid status on Data Center Default. Setting status to Non Responsive. >2014-08-07 14:37:35,378 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-20) [32f66d2c] hostFromVds::selectedVds - rack, spmStatus SPM, storage pool Default >2014-08-07 14:37:35,402 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-20) [32f66d2c] Initialize Irs proxy from vds: 10.34.62.208 >2014-08-07 14:37:35,419 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-20) [32f66d2c] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208). >2014-08-07 14:37:35,485 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-13) [32f66d2c] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 644cfc0f >2014-08-07 14:37:35,555 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-13) [32f66d2c] -- executeIrsBrokerCommand: Attempting on storage pool 00000002-0002-0002-0002-0000000001d9 >2014-08-07 14:37:35,571 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-13) [32f66d2c] START, HSMGetAllTasksInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 2ffedd7f >2014-08-07 14:37:35,581 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-13) [32f66d2c] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 2ffedd7f >2014-08-07 14:37:35,582 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-13) [32f66d2c] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 644cfc0f >2014-08-07 14:37:35,583 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (org.ovirt.thread.pool-8-thread-13) [32f66d2c] Discovered no tasks on Storage Pool Default >2014-08-07 14:37:50,503 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Loaded file "/usr/share/ovirt-engine/services/ovirt-engine/ovirt-engine.conf". >2014-08-07 14:37:50,504 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) The file "/etc/ovirt-engine/engine.conf" doesn't exist or isn't readable. Will return an empty set of properties. >2014-08-07 14:37:50,505 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Loaded file "/etc/ovirt-engine/engine.conf.d/10-setup-database.conf". >2014-08-07 14:37:50,505 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Loaded file "/etc/ovirt-engine/engine.conf.d/10-setup-jboss.conf". >2014-08-07 14:37:50,506 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Loaded file "/etc/ovirt-engine/engine.conf.d/10-setup-pki.conf". >2014-08-07 14:37:50,507 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Loaded file "/etc/ovirt-engine/engine.conf.d/10-setup-protocols.conf". >2014-08-07 14:37:50,507 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_AJP_ENABLED" is "true". >2014-08-07 14:37:50,517 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_AJP_PORT" is "8702". >2014-08-07 14:37:50,517 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_APPS" is "engine.ear restapi.war legacy_restapi.war". >2014-08-07 14:37:50,518 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_CACHE" is "/var/cache/ovirt-engine". >2014-08-07 14:37:50,518 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_DB_CHECK_INTERVAL" is "1000". >2014-08-07 14:37:50,518 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_DB_CONNECTION_TIMEOUT" is "300000". >2014-08-07 14:37:50,519 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_DB_DATABASE" is "engine". >2014-08-07 14:37:50,527 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_DB_DRIVER" is "org.postgresql.Driver". >2014-08-07 14:37:50,528 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_DB_HOST" is "localhost". >2014-08-07 14:37:50,528 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_DB_MAX_CONNECTIONS" is "100". >2014-08-07 14:37:50,529 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_DB_MIN_CONNECTIONS" is "1". >2014-08-07 14:37:50,529 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_DB_PASSWORD" is "***". >2014-08-07 14:37:50,530 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_DB_PORT" is "5432". >2014-08-07 14:37:50,530 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_DB_SECURED" is "False". >2014-08-07 14:37:50,534 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_DB_SECURED_VALIDATION" is "False". >2014-08-07 14:37:50,534 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_DB_URL" is "jdbc:postgresql://localhost:5432/engine?sslfactory=org.postgresql.ssl.NonValidatingFactory". >2014-08-07 14:37:50,535 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_DB_USER" is "engine". >2014-08-07 14:37:50,545 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_DEBUG_ADDRESS" is "". >2014-08-07 14:37:50,546 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_DOC" is "/usr/share/doc/ovirt-engine". >2014-08-07 14:37:50,546 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_ETC" is "/etc/ovirt-engine". >2014-08-07 14:37:50,547 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_EXTENSION_PATH" is "/usr/share/ovirt-engine/extensions.d:/etc/ovirt-engine/extensions.d". >2014-08-07 14:37:50,547 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_FQDN" is "pb-rh35.rhev.lab.eng.brq.redhat.com". >2014-08-07 14:37:50,548 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_GROUP" is "ovirt". >2014-08-07 14:37:50,548 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_HEAP_MAX" is "1g". >2014-08-07 14:37:50,556 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_HEAP_MIN" is "1g". >2014-08-07 14:37:50,556 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_HTTPS_ENABLED" is "false". >2014-08-07 14:37:50,557 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_HTTPS_PORT" is "None". >2014-08-07 14:37:50,558 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_HTTPS_PROTOCOLS" is "SSLv3,TLSv1,TLSv1.1,TLSv1.2". >2014-08-07 14:37:50,568 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_HTTP_ENABLED" is "false". >2014-08-07 14:37:50,568 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_HTTP_PORT" is "None". >2014-08-07 14:37:50,569 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_JAVA_MODULEPATH" is "/usr/share/ovirt-engine/modules". >2014-08-07 14:37:50,570 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_JVM_ARGS" is " -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath="/var/log/ovirt-engine/dump"". >2014-08-07 14:37:50,581 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_LOG" is "/var/log/ovirt-engine". >2014-08-07 14:37:50,581 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_LOG_TO_CONSOLE" is "false". >2014-08-07 14:37:50,582 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_MANUAL" is "/usr/share/ovirt-engine/manual". >2014-08-07 14:37:50,583 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_PERM_MAX" is "256m". >2014-08-07 14:37:50,584 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_PERM_MIN" is "256m". >2014-08-07 14:37:50,584 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_PKI" is "/etc/pki/ovirt-engine". >2014-08-07 14:37:50,597 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_PKI_CA" is "/etc/pki/ovirt-engine/ca.pem". >2014-08-07 14:37:50,598 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_PKI_ENGINE_CERT" is "/etc/pki/ovirt-engine/certs/engine.cer". >2014-08-07 14:37:50,599 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_PKI_ENGINE_STORE" is "/etc/pki/ovirt-engine/keys/engine.p12". >2014-08-07 14:37:50,608 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_PKI_ENGINE_STORE_ALIAS" is "1". >2014-08-07 14:37:50,609 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_PKI_ENGINE_STORE_PASSWORD" is "***". >2014-08-07 14:37:50,610 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_PKI_TRUST_STORE" is "/etc/pki/ovirt-engine/.truststore". >2014-08-07 14:37:50,611 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_PKI_TRUST_STORE_PASSWORD" is "***". >2014-08-07 14:37:50,611 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_PROPERTIES" is " jsse.enableSNIExtension=false "java.security.krb5.conf=/etc/ovirt-engine/krb5.conf"". >2014-08-07 14:37:50,620 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_PROXY_ENABLED" is "true". >2014-08-07 14:37:50,621 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_PROXY_HTTPS_PORT" is "443". >2014-08-07 14:37:50,622 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_PROXY_HTTP_PORT" is "80". >2014-08-07 14:37:50,622 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_REPORTS_UI" is "/var/lib/ovirt-engine/reports.xml". >2014-08-07 14:37:50,623 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_STOP_INTERVAL" is "1". >2014-08-07 14:37:50,624 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_STOP_TIME" is "10". >2014-08-07 14:37:50,632 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_TMP" is "/var/tmp/ovirt-engine". >2014-08-07 14:37:50,633 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_UP_MARK" is "/var/lib/ovirt-engine/engine.up". >2014-08-07 14:37:50,634 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_URI" is "/ovirt-engine". >2014-08-07 14:37:50,634 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_USER" is "ovirt". >2014-08-07 14:37:50,635 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_USR" is "/usr/share/ovirt-engine". >2014-08-07 14:37:50,644 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_VAR" is "/var/lib/ovirt-engine". >2014-08-07 14:37:50,645 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "ENGINE_VERBOSE_GC" is "false". >2014-08-07 14:37:50,645 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "JBOSS_HOME" is "/usr/share/ovirt-engine-jboss-as". >2014-08-07 14:37:50,646 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "JBOSS_RUNTIME" is "/var/lib/ovirt-engine/jboss_runtime". >2014-08-07 14:37:50,647 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-4) Value of property "SENSITIVE_KEYS" is ",ENGINE_DB_PASSWORD,ENGINE_PKI_TRUST_STORE_PASSWORD,ENGINE_PKI_ENGINE_STORE_PASSWORD". >2014-08-07 14:37:51,468 INFO [org.ovirt.engine.core.bll.Backend] (MSC service thread 1-4) Start initializing Backend >2014-08-07 14:37:51,911 INFO [org.ovirt.engine.core.utils.osinfo.OsInfoPreferencesLoader] (MSC service thread 1-4) Loaded file /etc/ovirt-engine/osinfo.conf.d/00-defaults.properties >2014-08-07 14:37:52,026 INFO [org.ovirt.engine.core.bll.Backend] (MSC service thread 1-4) Running ovirt-engine 3.5.0-0.0.master.20140804172041.git23b558e.el6 >2014-08-07 14:37:52,027 INFO [org.ovirt.engine.core.bll.CpuFlagsManagerHandler] (MSC service thread 1-4) Start initializing dictionaries >2014-08-07 14:37:52,034 INFO [org.ovirt.engine.core.bll.CpuFlagsManagerHandler] (MSC service thread 1-4) Finished initializing dictionaries >2014-08-07 14:37:52,034 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (MSC service thread 1-4) Start initializing AuditLogCleanupManager >2014-08-07 14:37:52,035 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (MSC service thread 1-4) Setting audit cleanup manager to run at: 35 35 3 * * ? >2014-08-07 14:37:52,056 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (MSC service thread 1-4) Finished initializing AuditLogCleanupManager >2014-08-07 14:37:52,057 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (MSC service thread 1-4) Start initializing CommandEntityCleanupManager >2014-08-07 14:37:52,057 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (MSC service thread 1-4) Setting command entity cleanup manager to run at: 35 35 3 * * ? >2014-08-07 14:37:52,062 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (MSC service thread 1-4) Finished initializing CommandEntityCleanupManager >2014-08-07 14:37:52,064 INFO [org.ovirt.engine.core.bll.TagsDirector] (MSC service thread 1-4) Start initializing TagsDirector >2014-08-07 14:37:52,085 INFO [org.ovirt.engine.core.bll.TagsDirector] (MSC service thread 1-4) Tag root added to tree >2014-08-07 14:37:52,104 INFO [org.ovirt.engine.core.bll.TagsDirector] (MSC service thread 1-4) Finished initializing TagsDirector >2014-08-07 14:37:52,105 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (MSC service thread 1-4) Start initializing IsoDomainListSyncronizer >2014-08-07 14:37:52,109 ERROR [org.ovirt.engine.core.aaa.filters.BasicAuthenticationFilter] (ajp--127.0.0.1-8702-8) Error in obtaining profile null >2014-08-07 14:37:52,117 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (MSC service thread 1-4) Finished initializing IsoDomainListSyncronizer >2014-08-07 14:37:52,203 INFO [org.ovirt.engine.core.bll.Backend] (MSC service thread 1-4) Completed initializing handlers >2014-08-07 14:37:52,230 INFO [org.ovirt.engine.core.utils.ErrorTranslatorImpl] (MSC service thread 1-4) Start initializing ErrorTranslatorImpl >2014-08-07 14:37:52,250 WARN [org.ovirt.engine.core.utils.ErrorTranslatorImpl] (MSC service thread 1-4) Code MAC_ADDRESS_IS_IN_USE appears more than once in string table. >2014-08-07 14:37:52,251 INFO [org.ovirt.engine.core.utils.ErrorTranslatorImpl] (MSC service thread 1-4) Finished initializing ErrorTranslatorImpl >2014-08-07 14:37:52,253 INFO [org.ovirt.engine.core.utils.ErrorTranslatorImpl] (MSC service thread 1-4) Start initializing ErrorTranslatorImpl >2014-08-07 14:37:52,254 INFO [org.ovirt.engine.core.utils.ErrorTranslatorImpl] (MSC service thread 1-4) Finished initializing ErrorTranslatorImpl >2014-08-07 14:37:52,255 INFO [org.ovirt.engine.core.bll.Backend] (MSC service thread 1-4) Mark incomplete jobs as UNKNOWN >2014-08-07 14:37:52,295 INFO [org.ovirt.engine.core.bll.job.JobRepositoryCleanupManager] (MSC service thread 1-4) Start initializing JobRepositoryCleanupManager >2014-08-07 14:37:52,296 INFO [org.ovirt.engine.core.bll.job.JobRepositoryCleanupManager] (MSC service thread 1-4) Finished initializing JobRepositoryCleanupManager >2014-08-07 14:37:52,296 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (MSC service thread 1-4) Start initializing AutoRecoveryManager >2014-08-07 14:37:52,299 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (MSC service thread 1-4) Finished initializing AutoRecoveryManager >2014-08-07 14:37:52,300 INFO [org.ovirt.engine.core.dal.job.ExecutionMessageDirector] (MSC service thread 1-4) Start initializing ExecutionMessageDirector >2014-08-07 14:37:52,304 INFO [org.ovirt.engine.core.dal.job.ExecutionMessageDirector] (MSC service thread 1-4) Finished initializing ExecutionMessageDirector >2014-08-07 14:37:52,364 INFO [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (MSC service thread 1-4) Start of enabled extensions list >2014-08-07 14:37:52,364 INFO [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (MSC service thread 1-4) Instance name: 'builtin-authn-internal', Extension name: 'Internal Authn (Built-in)', Version: 'N/A', Notes: '', License: 'ASL 2.0', Home: 'http://www.ovirt.org', Author 'The oVirt Project', Build interface Version: '0', File: 'N/A', Initialized: 'true' >2014-08-07 14:37:52,365 INFO [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (MSC service thread 1-4) Instance name: 'internal', Extension name: 'Internal Authz (Built-in)', Version: 'N/A', Notes: '', License: 'ASL 2.0', Home: 'http://www.ovirt.org', Author 'The oVirt Project', Build interface Version: '0', File: 'N/A', Initialized: 'true' >2014-08-07 14:37:52,367 INFO [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (MSC service thread 1-4) End of enabled extensions list >2014-08-07 14:37:52,469 INFO [org.ovirt.engine.core.bll.aaa.DbUserCacheManager] (MSC service thread 1-2) Start initializing DbUserCacheManager >2014-08-07 14:37:52,472 INFO [org.ovirt.engine.core.bll.aaa.DbUserCacheManager] (MSC service thread 1-2) Finished initializing DbUserCacheManager >2014-08-07 14:37:52,520 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (MSC service thread 1-2) Initialization of AsyncTaskManager completed successfully. >2014-08-07 14:37:52,522 INFO [org.ovirt.engine.core.vdsbroker.ResourceManager] (MSC service thread 1-2) Start initializing ResourceManager >2014-08-07 14:37:52,576 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-1) Principal admin::internal synchronized >2014-08-07 14:37:52,691 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (MSC service thread 1-2) Entered VdsManager constructor >2014-08-07 14:37:52,772 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (MSC service thread 1-2) Initialize vdsBroker (10.34.62.208,54,321) >2014-08-07 14:37:52,957 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-9) Running command: LoginUserCommand internal: false. >2014-08-07 14:37:52,969 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-9) Cannot convert the string Host ${VdsName} is not responding. It will stay in Connecting state for a grace period of ${Seconds} seconds and after that an attempt to fence the host will be issued. to AuditLogType, the key does not exist in the AuditLogType declared types >2014-08-07 14:37:52,975 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-9) AuditLogType: UNASSIGNED not exist in string table >2014-08-07 14:37:52,976 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-9) AuditLogType: VDS_AUTO_FENCE_STATUS not exist in string table >2014-08-07 14:37:52,976 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-9) AuditLogType: VDS_AUTO_FENCE_STATUS_FAILED not exist in string table >2014-08-07 14:37:52,977 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-9) AuditLogType: VDS_AUTO_FENCE_FAILED_CALL_FENCE_SPM not exist in string table >2014-08-07 14:37:52,978 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-9) AuditLogType: VDS_HIGH_NETWORK_USE not exist in string table >2014-08-07 14:37:52,980 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-9) AuditLogType: USER_FAILED_REMOVE_VM not exist in string table >2014-08-07 14:37:52,981 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-9) AuditLogType: USER_RUN_UNLOCK_ENTITY_SCRIPT not exist in string table >2014-08-07 14:37:52,982 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-9) AuditLogType: VDS_NETWORK_MTU_DIFFER_FROM_LOGICAL_NETWORK not exist in string table >2014-08-07 14:37:52,983 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-9) AuditLogType: VDS_HOST_IN_CONNECTING_STATE not exist in string table >2014-08-07 14:37:52,984 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-9) AuditLogType: STORAGE_ACTIVATE_ASYNC not exist in string table >2014-08-07 14:37:52,985 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-9) AuditLogType: DWH_STOPPED not exist in string table >2014-08-07 14:37:52,985 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-9) AuditLogType: DWH_STARTED not exist in string table >2014-08-07 14:37:52,986 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-9) AuditLogType: DWH_ERROR not exist in string table >2014-08-07 14:37:52,995 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-9) AuditLogType: USER_REMOVE_AUDIT_LOG not exist in string table >2014-08-07 14:37:52,996 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-9) AuditLogType: USER_REMOVE_AUDIT_LOG_FAILED not exist in string table >2014-08-07 14:37:52,997 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-9) AuditLogType: USER_CLEAR_ALL_DISMISSED_AUDIT_LOG not exist in string table >2014-08-07 14:37:52,998 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-9) AuditLogType: USER_CLEAR_ALL_DISMISSED_AUDIT_LOG_FAILED not exist in string table >2014-08-07 14:37:53,108 INFO [org.ovirt.engine.core.vdsbroker.ResourceManager] (MSC service thread 1-2) VDS 17f7fe56-5f32-40ef-b098-611b84484e0b was added to the Resource Manager >2014-08-07 14:37:53,151 INFO [org.ovirt.engine.core.vdsbroker.ResourceManager] (MSC service thread 1-2) Finished initializing ResourceManager >2014-08-07 14:37:53,162 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (MSC service thread 1-2) Initialization of OvfDataUpdater completed successfully. >2014-08-07 14:37:53,163 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (MSC service thread 1-2) Start scheduling to enable vds load balancer >2014-08-07 14:37:53,169 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (MSC service thread 1-2) Finished scheduling to enable vds load balancer >2014-08-07 14:37:53,170 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (MSC service thread 1-2) Start HA Reservation check >2014-08-07 14:37:53,171 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (MSC service thread 1-2) Finished HA Reservation check >2014-08-07 14:37:53,180 INFO [org.ovirt.engine.core.bll.network.MacPoolManagerRanges] (org.ovirt.thread.pool-8-thread-2) Start initializing MacPoolManagerRanges >2014-08-07 14:37:53,188 INFO [org.ovirt.engine.core.bll.InitBackendServicesOnStartupBean] (MSC service thread 1-2) Init VM custom properties utilities >2014-08-07 14:37:53,189 INFO [org.ovirt.engine.core.bll.InitBackendServicesOnStartupBean] (MSC service thread 1-2) Init device custom properties utilities >2014-08-07 14:37:53,197 INFO [org.ovirt.engine.core.bll.network.MacPoolManagerRanges] (org.ovirt.thread.pool-8-thread-2) Finished initializing. Available MACs in pool: 252 >2014-08-07 14:37:53,204 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (MSC service thread 1-2) Initializing Scheduling manager >2014-08-07 14:37:53,243 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (MSC service thread 1-2) External scheduler disabled, discovery skipped >2014-08-07 14:37:53,244 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (MSC service thread 1-2) Initialized Scheduling manager >2014-08-07 14:37:53,244 INFO [org.ovirt.engine.core.bll.dwh.DwhHeartBeat] (MSC service thread 1-2) Initializing DWH Heart Beat >2014-08-07 14:37:53,257 INFO [org.ovirt.engine.core.bll.dwh.DwhHeartBeat] (MSC service thread 1-2) DWH Heart Beat initialized >2014-08-07 14:37:53,322 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-9) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-08-07 14:37:56,778 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-6) START, GetHardwareInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]), log id: 3e13b940 >2014-08-07 14:37:56,833 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-6) FINISH, GetHardwareInfoVDSCommand, log id: 3e13b940 >2014-08-07 14:37:57,057 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-6) Initializing Host: rack >2014-08-07 14:37:57,080 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-6) [46247d52] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS >2014-08-07 14:38:03,241 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-19) [175991ea] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-08-07 14:38:03,264 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-19) [175991ea] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Active to Unknown, reason : null >2014-08-07 14:38:03,271 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-19) [175991ea] Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Active to Unknown, reason : null >2014-08-07 14:38:03,345 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-19) [175991ea] Correlation ID: 175991ea, Call Stack: null, Custom Event ID: -1, Message: Invalid status on Data Center Default. Setting status to Non Responsive. >2014-08-07 14:38:03,526 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-19) [175991ea] hostFromVds::selectedVds - rack, spmStatus SPM, storage pool Default >2014-08-07 14:38:03,537 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-19) [175991ea] Initialize Irs proxy from vds: 10.34.62.208 >2014-08-07 14:38:03,555 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-19) [175991ea] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208). >2014-08-07 14:38:03,593 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-15) [175991ea] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 52fc122e >2014-08-07 14:38:03,672 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-15) [175991ea] -- executeIrsBrokerCommand: Attempting on storage pool 00000002-0002-0002-0002-0000000001d9 >2014-08-07 14:38:03,685 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-15) [175991ea] START, HSMGetAllTasksInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 7d448d31 >2014-08-07 14:38:03,699 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-15) [175991ea] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 7d448d31 >2014-08-07 14:38:03,700 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-15) [175991ea] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 52fc122e >2014-08-07 14:38:03,702 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (org.ovirt.thread.pool-8-thread-15) [175991ea] Discovered no tasks on Storage Pool Default >2014-08-07 14:47:04,647 WARN [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-8-thread-35) Executing a command: java.util.concurrent.FutureTask , but note that there are 1 tasks in the queue. >2014-08-07 15:37:52,657 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-67) Principal admin::internal synchronized >2014-08-07 15:37:53,163 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-66) Attempting to update VMs/Templates Ovf. >2014-08-07 15:49:57,126 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-1) Running command: LoginUserCommand internal: false. >2014-08-07 15:49:57,133 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-1) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-08-07 15:49:57,155 INFO [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp--127.0.0.1-8702-1) [fcd8d62] Running command: LogoutBySessionCommand internal: false. >2014-08-07 15:49:57,183 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp--127.0.0.1-8702-1) [1615ec50] Running command: LogoutUserCommand internal: false. >2014-08-07 15:49:57,204 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-1) [1615ec50] Correlation ID: 1615ec50, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-08-07 15:49:59,294 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-3) Running command: LoginUserCommand internal: false. >2014-08-07 15:50:01,733 INFO [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp--127.0.0.1-8702-3) [5b9c38ac] Running command: LogoutBySessionCommand internal: false. >2014-08-07 15:50:01,736 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp--127.0.0.1-8702-3) [1b0b2960] Running command: LogoutUserCommand internal: false. >2014-08-07 15:50:01,740 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-3) [1b0b2960] Correlation ID: 1b0b2960, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-08-07 15:54:30,563 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-5) Running command: LoginUserCommand internal: false. >2014-08-07 15:54:30,570 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-5) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-08-07 15:54:30,582 INFO [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp--127.0.0.1-8702-5) [262446e2] Running command: LogoutBySessionCommand internal: false. >2014-08-07 15:54:30,587 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp--127.0.0.1-8702-5) [7e2285fd] Running command: LogoutUserCommand internal: false. >2014-08-07 15:54:30,596 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-5) [7e2285fd] Correlation ID: 7e2285fd, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-08-07 15:54:32,230 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-7) Running command: LoginUserCommand internal: false. >2014-08-07 15:54:32,798 INFO [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp--127.0.0.1-8702-7) [633648f5] Running command: LogoutBySessionCommand internal: false. >2014-08-07 15:54:32,802 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp--127.0.0.1-8702-7) [1b94efa] Running command: LogoutUserCommand internal: false. >2014-08-07 15:54:32,806 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-7) [1b94efa] Correlation ID: 1b94efa, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-08-07 15:55:38,518 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-9) Running command: LoginUserCommand internal: false. >2014-08-07 15:55:38,524 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-9) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-08-07 15:55:38,539 INFO [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp--127.0.0.1-8702-9) [6862e83] Running command: LogoutBySessionCommand internal: false. >2014-08-07 15:55:38,545 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp--127.0.0.1-8702-9) [69e17b3b] Running command: LogoutUserCommand internal: false. >2014-08-07 15:55:38,552 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-9) [69e17b3b] Correlation ID: 69e17b3b, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-08-07 15:55:39,899 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-11) Running command: LoginUserCommand internal: false. >2014-08-07 15:55:40,328 INFO [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp--127.0.0.1-8702-11) [77f231bd] Running command: LogoutBySessionCommand internal: false. >2014-08-07 15:55:40,332 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp--127.0.0.1-8702-11) [78465d1d] Running command: LogoutUserCommand internal: false. >2014-08-07 15:55:40,337 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-11) [78465d1d] Correlation ID: 78465d1d, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-08-07 16:04:52,004 INFO [org.ovirt.engine.core.bll.aaa.LoginAdminUserCommand] (ajp--127.0.0.1-8702-1) Running command: LoginAdminUserCommand internal: false. >2014-08-07 16:04:52,012 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-1) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-08-07 16:04:53,815 INFO [org.ovirt.engine.core.bll.aaa.LoginAdminUserCommand] (ajp--127.0.0.1-8702-2) Running command: LoginAdminUserCommand internal: false. >2014-08-07 16:04:59,089 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-6) Running command: LoginUserCommand internal: false. >2014-08-07 16:04:59,094 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-08-07 16:05:32,225 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-10) Running command: LoginUserCommand internal: false. >2014-08-07 16:05:32,229 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-10) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-08-07 16:05:32,238 INFO [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp--127.0.0.1-8702-10) [50490546] Running command: LogoutBySessionCommand internal: false. >2014-08-07 16:05:32,247 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp--127.0.0.1-8702-10) [ae2fc2] Running command: LogoutUserCommand internal: false. >2014-08-07 16:05:32,257 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-10) [ae2fc2] Correlation ID: ae2fc2, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-08-07 16:05:34,047 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-2) Running command: LoginUserCommand internal: false. >2014-08-07 16:05:34,392 INFO [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp--127.0.0.1-8702-2) [3242db73] Running command: LogoutBySessionCommand internal: false. >2014-08-07 16:05:34,400 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp--127.0.0.1-8702-2) [7bbf4404] Running command: LogoutUserCommand internal: false. >2014-08-07 16:05:34,404 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-2) [7bbf4404] Correlation ID: 7bbf4404, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-08-07 16:27:31,622 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-3) Running command: LoginUserCommand internal: false. >2014-08-07 16:27:31,629 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-3) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-08-07 16:27:31,636 INFO [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp--127.0.0.1-8702-3) [35895a] Running command: LogoutBySessionCommand internal: false. >2014-08-07 16:27:31,640 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp--127.0.0.1-8702-3) [3098c960] Running command: LogoutUserCommand internal: false. >2014-08-07 16:27:31,647 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-3) [3098c960] Correlation ID: 3098c960, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-08-07 16:27:32,949 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-4) Running command: LoginUserCommand internal: false. >2014-08-07 16:27:33,440 INFO [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp--127.0.0.1-8702-4) [574a1f51] Running command: LogoutBySessionCommand internal: false. >2014-08-07 16:27:33,443 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp--127.0.0.1-8702-4) [4a999e1e] Running command: LogoutUserCommand internal: false. >2014-08-07 16:27:33,448 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-4) [4a999e1e] Correlation ID: 4a999e1e, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-08-07 16:37:52,669 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-47) Principal admin::internal synchronized >2014-08-07 16:37:53,592 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-67) Attempting to update VMs/Templates Ovf. >2014-08-07 16:57:20,044 INFO [org.ovirt.engine.core.bll.storage.AttachStorageDomainToPoolCommand] (org.ovirt.thread.pool-8-thread-22) [277d6f20] Running command: AttachStorageDomainToPoolCommand internal: false. Entities affected : ID: 63fe3516-4170-402a-9dc8-6d3eab1c7933 Type: Storage >2014-08-07 16:57:20,140 INFO [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-8-thread-42) [2e959e56] Running command: ConnectStorageToVdsCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2014-08-07 16:57:20,150 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-42) [2e959e56] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: 7fe036a2-2dfc-4565-b936-69a678cf0613, connection: pb-rh35.rhev.lab.eng.brq.redhat.com:/var/lib/exports/iso, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: caab5d3 >2014-08-07 16:59:30,285 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-42) [2e959e56] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host rack to the Storage Domains ISO_DOMAIN. >2014-08-07 16:59:30,287 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-42) [2e959e56] FINISH, ConnectStorageServerVDSCommand, return: {7fe036a2-2dfc-4565-b936-69a678cf0613=477}, log id: caab5d3 >2014-08-07 16:59:30,291 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-42) [2e959e56] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: The error message for connection pb-rh35.rhev.lab.eng.brq.redhat.com:/var/lib/exports/iso returned by VDSM was: Problem while trying to mount target >2014-08-07 16:59:30,295 ERROR [org.ovirt.engine.core.bll.storage.NFSStorageHelper] (org.ovirt.thread.pool-8-thread-42) [2e959e56] The connection with details pb-rh35.rhev.lab.eng.brq.redhat.com:/var/lib/exports/iso failed because of error code 477 and error message is: problem while trying to mount target >2014-08-07 16:59:30,318 ERROR [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-8-thread-42) [2e959e56] Transaction rolled-back for command: org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand. >2014-08-07 16:59:30,329 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.AttachStorageDomainVDSCommand] (org.ovirt.thread.pool-8-thread-22) [277d6f20] START, AttachStorageDomainVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, storageDomainId = 63fe3516-4170-402a-9dc8-6d3eab1c7933), log id: 5dfbba80 >2014-08-07 17:01:30,864 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.AttachStorageDomainVDSCommand] (org.ovirt.thread.pool-8-thread-22) [277d6f20] Failed in AttachStorageDomainVDS method >2014-08-07 17:01:30,920 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (org.ovirt.thread.pool-8-thread-22) [277d6f20] IrsBroker::Failed::AttachStorageDomainVDS due to: IRSErrorException: IRSGenericException: IRSErrorException: Failed to AttachStorageDomainVDS, error = Storage domain does not exist: ('63fe3516-4170-402a-9dc8-6d3eab1c7933',), code = 358 >2014-08-07 17:01:30,935 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.AttachStorageDomainVDSCommand] (org.ovirt.thread.pool-8-thread-22) [277d6f20] FINISH, AttachStorageDomainVDSCommand, log id: 5dfbba80 >2014-08-07 17:01:30,939 ERROR [org.ovirt.engine.core.bll.storage.AttachStorageDomainToPoolCommand] (org.ovirt.thread.pool-8-thread-22) [277d6f20] Command org.ovirt.engine.core.bll.storage.AttachStorageDomainToPoolCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.irsbroker.IRSErrorException: IRSGenericException: IRSErrorException: Failed to AttachStorageDomainVDS, error = Storage domain does not exist: ('63fe3516-4170-402a-9dc8-6d3eab1c7933',), code = 358 (Failed with error StorageDomainDoesNotExist and code 358) >2014-08-07 17:01:30,994 INFO [org.ovirt.engine.core.bll.storage.AttachStorageDomainToPoolCommand] (org.ovirt.thread.pool-8-thread-22) [277d6f20] Command [id=cb7ecc9d-6a6a-41e9-bb28-b5694d83abb7]: Compensating NEW_ENTITY_ID of org.ovirt.engine.core.common.businessentities.StoragePoolIsoMap; snapshot: storagePoolId = 00000002-0002-0002-0002-0000000001d9, storageId = 63fe3516-4170-402a-9dc8-6d3eab1c7933. >2014-08-07 17:01:31,031 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-22) [277d6f20] Correlation ID: 277d6f20, Job ID: d1972209-3011-4316-9e81-c90bf97a25f9, Call Stack: null, Custom Event ID: -1, Message: Failed to attach Storage Domain ISO_DOMAIN to Data Center Default. (User: admin) >2014-08-07 17:13:26,085 INFO [org.ovirt.engine.core.bll.storage.AttachStorageDomainToPoolCommand] (org.ovirt.thread.pool-8-thread-21) [225a8704] Running command: AttachStorageDomainToPoolCommand internal: false. Entities affected : ID: 63fe3516-4170-402a-9dc8-6d3eab1c7933 Type: Storage >2014-08-07 17:13:26,099 INFO [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-8-thread-6) [367e5883] Running command: ConnectStorageToVdsCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2014-08-07 17:13:26,105 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-6) [367e5883] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: 7fe036a2-2dfc-4565-b936-69a678cf0613, connection: pb-rh35.rhev.lab.eng.brq.redhat.com:/var/lib/exports/iso, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 22ba5258 >2014-08-07 17:14:28,971 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-6) [367e5883] FINISH, ConnectStorageServerVDSCommand, return: {7fe036a2-2dfc-4565-b936-69a678cf0613=0}, log id: 22ba5258 >2014-08-07 17:14:28,976 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.AttachStorageDomainVDSCommand] (org.ovirt.thread.pool-8-thread-21) [225a8704] START, AttachStorageDomainVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, storageDomainId = 63fe3516-4170-402a-9dc8-6d3eab1c7933), log id: 6814f7d7 >2014-08-07 17:14:43,317 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.AttachStorageDomainVDSCommand] (org.ovirt.thread.pool-8-thread-21) [225a8704] FINISH, AttachStorageDomainVDSCommand, log id: 6814f7d7 >2014-08-07 17:14:43,329 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetImagesListVDSCommand] (org.ovirt.thread.pool-8-thread-21) [225a8704] START, GetImagesListVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, sdUUID = 63fe3516-4170-402a-9dc8-6d3eab1c7933), log id: 7601da2 >2014-08-07 17:14:43,436 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetImagesListVDSCommand] (org.ovirt.thread.pool-8-thread-21) [225a8704] FINISH, GetImagesListVDSCommand, return: [11111111-1111-1111-1111-111111111111], log id: 7601da2 >2014-08-07 17:14:43,462 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetVolumesListVDSCommand] (org.ovirt.thread.pool-8-thread-21) [225a8704] START, GetVolumesListVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, storageDomainId = 63fe3516-4170-402a-9dc8-6d3eab1c7933, imageGroupId = 11111111-1111-1111-1111-111111111111), log id: 7dd47f5b >2014-08-07 17:14:43,512 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetVolumesListVDSCommand] (org.ovirt.thread.pool-8-thread-21) [225a8704] FINISH, GetVolumesListVDSCommand, return: [], log id: 7dd47f5b >2014-08-07 17:14:43,513 ERROR [org.ovirt.engine.core.bll.GetUnregisteredDisksQuery] (org.ovirt.thread.pool-8-thread-21) [225a8704] Could not get populated disk, reason: null >2014-08-07 17:14:43,515 WARN [org.ovirt.engine.core.bll.storage.AttachStorageDomainToPoolCommand] (org.ovirt.thread.pool-8-thread-21) There are no OVF_STORE disks on storage domain id 63fe3516-4170-402a-9dc8-6d3eab1c7933 >2014-08-07 17:14:43,523 INFO [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-21) [948eec4] Lock Acquired to object EngineLock [exclusiveLocks= key: 63fe3516-4170-402a-9dc8-6d3eab1c7933 value: STORAGE >, sharedLocks= ] >2014-08-07 17:14:43,544 INFO [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-21) [948eec4] Running command: ActivateStorageDomainCommand internal: true. Entities affected : ID: 63fe3516-4170-402a-9dc8-6d3eab1c7933 Type: Storage >2014-08-07 17:14:43,559 INFO [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-21) [948eec4] Lock freed to object EngineLock [exclusiveLocks= key: 63fe3516-4170-402a-9dc8-6d3eab1c7933 value: STORAGE >, sharedLocks= ] >2014-08-07 17:14:43,561 INFO [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-21) [948eec4] ActivateStorage Domain. Before Connect all hosts to pool. Time:8/7/14 5:14 PM >2014-08-07 17:14:43,567 INFO [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-8-thread-5) [738f4946] Running command: ConnectStorageToVdsCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2014-08-07 17:14:43,574 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-5) [738f4946] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: 7fe036a2-2dfc-4565-b936-69a678cf0613, connection: pb-rh35.rhev.lab.eng.brq.redhat.com:/var/lib/exports/iso, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 454862f3 >2014-08-07 17:14:43,589 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-5) [738f4946] FINISH, ConnectStorageServerVDSCommand, return: {7fe036a2-2dfc-4565-b936-69a678cf0613=0}, log id: 454862f3 >2014-08-07 17:14:43,592 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ActivateStorageDomainVDSCommand] (org.ovirt.thread.pool-8-thread-21) [948eec4] START, ActivateStorageDomainVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, storageDomainId = 63fe3516-4170-402a-9dc8-6d3eab1c7933), log id: 5f634ce9 >2014-08-07 17:14:43,743 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ActivateStorageDomainVDSCommand] (org.ovirt.thread.pool-8-thread-21) [948eec4] FINISH, ActivateStorageDomainVDSCommand, log id: 5f634ce9 >2014-08-07 17:14:43,744 INFO [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-21) [948eec4] ActivateStorage Domain. After Connect all hosts to pool. Time:8/7/14 5:14 PM >2014-08-07 17:14:43,763 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-15) START, ConnectStoragePoolVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, masterVersion = 1), log id: 2c46611a >2014-08-07 17:14:44,088 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-15) FINISH, ConnectStoragePoolVDSCommand, log id: 2c46611a >2014-08-07 17:14:44,089 INFO [org.ovirt.engine.core.bll.storage.RefreshPoolSingleAsyncOperation] (org.ovirt.thread.pool-8-thread-15) Refreshed vds rack in pool Default >2014-08-07 17:14:44,090 INFO [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-21) [948eec4] ActivateStorage Domain. After change storage pool status in vds. Time:8/7/14 5:14 PM >2014-08-07 17:14:44,095 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-18) [948eec4] START, GetFileStatsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 4829e40a >2014-08-07 17:14:44,131 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-21) [948eec4] Correlation ID: 948eec4, Call Stack: null, Custom Event ID: -1, Message: Storage Domain ISO_DOMAIN (Data Center Default) was activated by <UNKNOWN> >2014-08-07 17:14:44,139 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-21) [948eec4] Correlation ID: 225a8704, Job ID: 71065b7b-b89f-46d4-82db-c8523e67889c, Call Stack: null, Custom Event ID: -1, Message: Storage Domain ISO_DOMAIN was attached to Data Center Default by admin >2014-08-07 17:14:44,156 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-18) [948eec4] FINISH, GetFileStatsVDSCommand, return: {}, log id: 4829e40a >2014-08-07 17:14:44,167 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-18) [948eec4] START, GetFileStatsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 27bd6ccf >2014-08-07 17:14:44,216 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-18) [948eec4] FINISH, GetFileStatsVDSCommand, return: {}, log id: 27bd6ccf >2014-08-07 17:37:52,676 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-16) Principal admin::internal synchronized >2014-08-07 17:37:53,636 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-33) Attempting to update VMs/Templates Ovf. >2014-08-07 17:40:58,795 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-5) Running command: LoginUserCommand internal: false. >2014-08-07 17:40:58,801 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-5) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-08-07 17:41:00,627 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (ajp--127.0.0.1-8702-6) START, GetFileStatsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 3df34cd7 >2014-08-07 17:41:00,643 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (ajp--127.0.0.1-8702-6) FINISH, GetFileStatsVDSCommand, return: {test.iso={status=0, ctime=1407430000.0, size=4}}, log id: 3df34cd7 >2014-08-07 17:41:00,651 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (ajp--127.0.0.1-8702-6) START, GetFileStatsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 5e1c7be7 >2014-08-07 17:41:00,701 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (ajp--127.0.0.1-8702-6) FINISH, GetFileStatsVDSCommand, return: {}, log id: 5e1c7be7 >2014-08-07 17:41:00,707 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Refresh image list succeeded for domain(s): ISO_DOMAIN (All file type) >2014-08-07 17:45:32,882 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-11) Running command: LoginUserCommand internal: false. >2014-08-07 17:45:32,889 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-11) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-08-07 17:45:32,955 INFO [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp--127.0.0.1-8702-11) [74d3cf3] Running command: LogoutBySessionCommand internal: false. >2014-08-07 17:45:32,960 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp--127.0.0.1-8702-11) [221af9ce] Running command: LogoutUserCommand internal: false. >2014-08-07 17:45:32,965 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-11) [221af9ce] Correlation ID: 221af9ce, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-08-07 17:45:33,412 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-9) Running command: LoginUserCommand internal: false. >2014-08-07 17:45:33,485 INFO [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp--127.0.0.1-8702-9) [2e06c8ab] Running command: LogoutBySessionCommand internal: false. >2014-08-07 17:45:33,490 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp--127.0.0.1-8702-9) [8897b51] Running command: LogoutUserCommand internal: false. >2014-08-07 17:45:33,496 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-9) [8897b51] Correlation ID: 8897b51, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-08-07 17:45:57,952 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-2) Running command: LoginUserCommand internal: false. >2014-08-07 17:45:57,956 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-2) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-08-07 17:45:58,788 INFO [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp--127.0.0.1-8702-2) [44406a47] Running command: LogoutBySessionCommand internal: false. >2014-08-07 17:45:58,791 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp--127.0.0.1-8702-2) [11e60992] Running command: LogoutUserCommand internal: false. >2014-08-07 17:45:58,795 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-2) [11e60992] Correlation ID: 11e60992, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-08-07 17:48:30,780 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-6) Running command: LoginUserCommand internal: false. >2014-08-07 17:48:30,786 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-08-07 17:48:31,611 INFO [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp--127.0.0.1-8702-6) [58001122] Running command: LogoutBySessionCommand internal: false. >2014-08-07 17:48:31,615 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp--127.0.0.1-8702-6) [6f5d23b2] Running command: LogoutUserCommand internal: false. >2014-08-07 17:48:31,622 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) [6f5d23b2] Correlation ID: 6f5d23b2, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-08-07 17:48:51,350 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-11) Running command: LoginUserCommand internal: false. >2014-08-07 17:48:51,352 WARN [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-8-thread-43) Executing a command: java.util.concurrent.FutureTask , but note that there are 1 tasks in the queue. >2014-08-07 17:48:51,354 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-11) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-08-07 17:48:51,394 INFO [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp--127.0.0.1-8702-11) [58459177] Running command: LogoutBySessionCommand internal: false. >2014-08-07 17:48:51,397 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp--127.0.0.1-8702-11) [36dcd673] Running command: LogoutUserCommand internal: false. >2014-08-07 17:48:51,406 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-11) [36dcd673] Correlation ID: 36dcd673, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-08-07 17:49:15,882 WARN [org.ovirt.engine.core.dal.job.ExecutionMessageDirector] (ajp--127.0.0.1-8702-7) [493d107a] The message key AddBookmark is missing from bundles/ExecutionMessages >2014-08-07 17:49:15,916 INFO [org.ovirt.engine.core.bll.AddBookmarkCommand] (ajp--127.0.0.1-8702-7) [493d107a] Running command: AddBookmarkCommand internal: false. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2014-08-07 17:49:15,943 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-7) [493d107a] Correlation ID: 493d107a, Job ID: dc802394-6944-4a31-8869-e2216492ff43, Call Stack: null, Custom Event ID: -1, Message: Bookmark test was added by admin. >2014-08-07 17:49:25,251 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-5) Running command: LoginUserCommand internal: false. >2014-08-07 17:49:25,256 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-5) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-08-07 17:49:25,266 INFO [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp--127.0.0.1-8702-5) [6ace6da3] Running command: LogoutBySessionCommand internal: false. >2014-08-07 17:49:25,270 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp--127.0.0.1-8702-5) [43a1c18f] Running command: LogoutUserCommand internal: false. >2014-08-07 17:49:25,275 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-5) [43a1c18f] Correlation ID: 43a1c18f, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-08-07 17:50:01,784 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-8) Running command: LoginUserCommand internal: false. >2014-08-07 17:50:01,795 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-8) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-08-07 17:50:03,150 INFO [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp--127.0.0.1-8702-8) [1c6b0ab1] Running command: LogoutBySessionCommand internal: false. >2014-08-07 17:50:03,154 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp--127.0.0.1-8702-8) [f349986] Running command: LogoutUserCommand internal: false. >2014-08-07 17:50:03,159 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-8) [f349986] Correlation ID: f349986, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-08-07 17:50:22,035 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-11) Running command: LoginUserCommand internal: false. >2014-08-07 17:50:22,049 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-11) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-08-07 17:50:22,080 WARN [org.ovirt.engine.core.dal.job.ExecutionMessageDirector] (ajp--127.0.0.1-8702-11) [32e0de7] The message key AddBookmark is missing from bundles/ExecutionMessages >2014-08-07 17:50:22,110 INFO [org.ovirt.engine.core.bll.AddBookmarkCommand] (ajp--127.0.0.1-8702-11) [32e0de7] Running command: AddBookmarkCommand internal: false. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2014-08-07 17:50:22,125 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-11) [32e0de7] Correlation ID: 32e0de7, Job ID: 573c6899-eb68-4072-93f9-5cfe67db3be4, Call Stack: null, Custom Event ID: -1, Message: Bookmark foo was added by admin. >2014-08-07 17:50:22,985 INFO [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp--127.0.0.1-8702-11) [1a4b6286] Running command: LogoutBySessionCommand internal: false. >2014-08-07 17:50:22,988 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp--127.0.0.1-8702-11) [5f1cea48] Running command: LogoutUserCommand internal: false. >2014-08-07 17:50:22,994 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-11) [5f1cea48] Correlation ID: 5f1cea48, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-08-07 17:50:31,646 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-3) Running command: LoginUserCommand internal: false. >2014-08-07 17:50:31,650 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-3) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-08-07 17:50:31,665 INFO [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp--127.0.0.1-8702-3) [613224bf] Running command: LogoutBySessionCommand internal: false. >2014-08-07 17:50:31,668 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp--127.0.0.1-8702-3) [8fe854e] Running command: LogoutUserCommand internal: false. >2014-08-07 17:50:31,677 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-3) [8fe854e] Correlation ID: 8fe854e, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-08-07 17:50:41,863 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-10) Running command: LoginUserCommand internal: false. >2014-08-07 17:50:41,868 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-10) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-08-07 17:50:41,894 WARN [org.ovirt.engine.core.dal.job.ExecutionMessageDirector] (ajp--127.0.0.1-8702-10) [2a4e5287] The message key RemoveBookmark is missing from bundles/ExecutionMessages >2014-08-07 17:50:41,921 INFO [org.ovirt.engine.core.bll.RemoveBookmarkCommand] (ajp--127.0.0.1-8702-10) [2a4e5287] Running command: RemoveBookmarkCommand internal: false. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2014-08-07 17:50:41,945 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-10) [2a4e5287] Correlation ID: 2a4e5287, Job ID: cb44395a-ed6d-4fce-a592-18fb9f5fad68, Call Stack: null, Custom Event ID: -1, Message: Bookmark test was removed by admin. >2014-08-07 17:50:42,677 INFO [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp--127.0.0.1-8702-10) [2d7f9d31] Running command: LogoutBySessionCommand internal: false. >2014-08-07 17:50:42,681 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp--127.0.0.1-8702-10) [66f6c478] Running command: LogoutUserCommand internal: false. >2014-08-07 17:50:42,684 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-10) [66f6c478] Correlation ID: 66f6c478, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-08-07 17:50:58,098 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-12) Running command: LoginUserCommand internal: false. >2014-08-07 17:50:58,102 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-12) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-08-07 17:50:58,117 INFO [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp--127.0.0.1-8702-12) [10a292e2] Running command: LogoutBySessionCommand internal: false. >2014-08-07 17:50:58,126 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp--127.0.0.1-8702-12) [61d7a27c] Running command: LogoutUserCommand internal: false. >2014-08-07 17:50:58,139 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-12) [61d7a27c] Correlation ID: 61d7a27c, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-08-07 17:51:42,594 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-6) Running command: LoginUserCommand internal: false. >2014-08-07 17:51:42,598 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-08-07 17:51:42,624 WARN [org.ovirt.engine.core.dal.job.ExecutionMessageDirector] (ajp--127.0.0.1-8702-6) [62803950] The message key UpdateBookmark is missing from bundles/ExecutionMessages >2014-08-07 17:51:42,650 INFO [org.ovirt.engine.core.bll.UpdateBookmarkCommand] (ajp--127.0.0.1-8702-6) [62803950] Running command: UpdateBookmarkCommand internal: false. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2014-08-07 17:51:42,663 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) [62803950] Correlation ID: 62803950, Job ID: 9341c107-be35-45a2-9e3b-d4ac4cbca465, Call Stack: null, Custom Event ID: -1, Message: Bookmark foo was updated by admin. >2014-08-07 17:51:42,675 INFO [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp--127.0.0.1-8702-6) [13c7ca57] Running command: LogoutBySessionCommand internal: false. >2014-08-07 17:51:42,680 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp--127.0.0.1-8702-6) [17034c5] Running command: LogoutUserCommand internal: false. >2014-08-07 17:51:42,685 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) [17034c5] Correlation ID: 17034c5, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-08-07 17:52:04,607 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-8) Running command: LoginUserCommand internal: false. >2014-08-07 17:52:04,612 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-8) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-08-07 17:52:04,618 INFO [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp--127.0.0.1-8702-8) [753c9bb0] Running command: LogoutBySessionCommand internal: false. >2014-08-07 17:52:04,621 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp--127.0.0.1-8702-8) [3972eaff] Running command: LogoutUserCommand internal: false. >2014-08-07 17:52:04,626 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-8) [3972eaff] Correlation ID: 3972eaff, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-08-07 17:53:14,908 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-10) Running command: LoginUserCommand internal: false. >2014-08-07 17:53:14,911 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-10) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-08-07 17:53:14,972 INFO [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp--127.0.0.1-8702-10) [3d8faed7] Running command: LogoutBySessionCommand internal: false. >2014-08-07 17:53:14,977 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp--127.0.0.1-8702-10) [6bd2b9a7] Running command: LogoutUserCommand internal: false. >2014-08-07 17:53:14,981 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-10) [6bd2b9a7] Correlation ID: 6bd2b9a7, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-08-07 17:54:13,534 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-7) Running command: LoginUserCommand internal: false. >2014-08-07 17:54:13,539 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-7) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-08-07 17:54:14,348 INFO [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp--127.0.0.1-8702-7) [102dcf63] Running command: LogoutBySessionCommand internal: false. >2014-08-07 17:54:14,352 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp--127.0.0.1-8702-7) [6f61bf18] Running command: LogoutUserCommand internal: false. >2014-08-07 17:54:14,356 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-7) [6f61bf18] Correlation ID: 6f61bf18, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-08-07 17:54:50,572 INFO [org.ovirt.engine.core.bll.AddVmTemplateCommand] (ajp--127.0.0.1-8702-9) Running command: AddVmTemplateCommand internal: false. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-08-07 17:54:50,582 INFO [org.ovirt.engine.core.vdsbroker.SetVmStatusVDSCommand] (ajp--127.0.0.1-8702-9) START, SetVmStatusVDSCommand( vmId = 9042bb21-3f27-48bb-ab26-c0d8b18c840d, status = ImageLocked, exit status = Normal), log id: 3f908ee5 >2014-08-07 17:54:50,597 INFO [org.ovirt.engine.core.vdsbroker.SetVmStatusVDSCommand] (ajp--127.0.0.1-8702-9) FINISH, SetVmStatusVDSCommand, log id: 3f908ee5 >2014-08-07 17:54:50,652 INFO [org.ovirt.engine.core.bll.CreateImageTemplateCommand] (ajp--127.0.0.1-8702-9) [63c518d5] Running command: CreateImageTemplateCommand internal: true. Entities affected : ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage >2014-08-07 17:54:50,679 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.CopyImageVDSCommand] (ajp--127.0.0.1-8702-9) [63c518d5] START, CopyImageVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, storageDomainId = 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752, imageGroupId = cc7b5e15-0557-460a-af76-7daa89c27d9f, imageId = 1efc6019-96fd-4b59-8289-6ce7fe14a614, dstImageGroupId = a5dac258-8f4f-48ae-b8a0-1e58a6b190db, vmId = 9042bb21-3f27-48bb-ab26-c0d8b18c840d, dstImageId = 46253726-8595-4414-b570-faf47fd84c4f, imageDescription = Active VM, dstStorageDomainId = 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752, copyVolumeType = SharedVol, volumeFormat = RAW, preallocate = Sparse, postZero = false, force = false), log id: 7880f82a >2014-08-07 17:54:50,689 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.CopyImageVDSCommand] (ajp--127.0.0.1-8702-9) [63c518d5] -- executeIrsBrokerCommand: calling 'copyImage' with two new parameters: description and UUID >2014-08-07 17:54:50,690 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.CopyImageVDSCommand] (ajp--127.0.0.1-8702-9) [63c518d5] -- copyImage parameters: > sdUUID=71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 > spUUID=00000002-0002-0002-0002-0000000001d9 > vmGUID=9042bb21-3f27-48bb-ab26-c0d8b18c840d > srcImageGUID=cc7b5e15-0557-460a-af76-7daa89c27d9f > srcVolUUID=1efc6019-96fd-4b59-8289-6ce7fe14a614 > dstImageGUID=a5dac258-8f4f-48ae-b8a0-1e58a6b190db > dstVolUUID=46253726-8595-4414-b570-faf47fd84c4f > descr=Active VM > dstSdUUID=71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 >2014-08-07 17:54:51,214 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.CopyImageVDSCommand] (ajp--127.0.0.1-8702-9) [63c518d5] FINISH, CopyImageVDSCommand, return: 00000000-0000-0000-0000-000000000000, log id: 7880f82a >2014-08-07 17:54:51,264 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (ajp--127.0.0.1-8702-9) [63c518d5] CommandAsyncTask::Adding CommandMultiAsyncTasks object for command ecc942ac-98b9-4e7a-9ff7-ac1229ca1a7c >2014-08-07 17:54:51,265 INFO [org.ovirt.engine.core.bll.CommandMultiAsyncTasks] (ajp--127.0.0.1-8702-9) [63c518d5] CommandMultiAsyncTasks::AttachTask: Attaching task 0e65f6c1-4537-402b-b37f-abecd8640972 to command ecc942ac-98b9-4e7a-9ff7-ac1229ca1a7c. >2014-08-07 17:54:51,281 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (ajp--127.0.0.1-8702-9) [63c518d5] Adding task 0e65f6c1-4537-402b-b37f-abecd8640972 (Parent Command AddVmTemplate, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters), polling hasn't started yet.. >2014-08-07 17:54:51,365 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-9) Correlation ID: 12896a2a, Job ID: 9aeb32de-ebca-494d-b093-f812995847a0, Call Stack: null, Custom Event ID: -1, Message: Creation of Template blah from VM ovirtnode was initiated by admin. >2014-08-07 17:54:51,366 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (ajp--127.0.0.1-8702-9) BaseAsyncTask::startPollingTask: Starting to poll task 0e65f6c1-4537-402b-b37f-abecd8640972. >2014-08-07 17:54:53,160 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (DefaultQuartzScheduler_Worker-56) Polling and updating Async Tasks: 1 tasks, 1 tasks to poll now >2014-08-07 17:54:53,173 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (DefaultQuartzScheduler_Worker-56) SPMAsyncTask::PollTask: Polling task 0e65f6c1-4537-402b-b37f-abecd8640972 (Parent Command AddVmTemplate, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) returned status running. >2014-08-07 17:54:53,188 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (DefaultQuartzScheduler_Worker-56) Finished polling Tasks, will poll again in 10 seconds. >2014-08-07 17:55:34,471 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-60) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host rack has network interface which exceeded the defined threshold [95%] (em1: transmit rate[0%], receive rate [98%]) >2014-08-07 17:56:41,109 WARN [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-8-thread-47) Executing a command: java.util.concurrent.FutureTask , but note that there are 1 tasks in the queue. >2014-08-07 17:56:41,178 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (DefaultQuartzScheduler_Worker-35) SPMAsyncTask::PollTask: Polling task 0e65f6c1-4537-402b-b37f-abecd8640972 (Parent Command AddVmTemplate, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) returned status finished, result 'success'. >2014-08-07 17:56:41,189 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (DefaultQuartzScheduler_Worker-35) BaseAsyncTask::OnTaskEndSuccess: Task 0e65f6c1-4537-402b-b37f-abecd8640972 (Parent Command AddVmTemplate, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) ended successfully. >2014-08-07 17:56:41,192 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (DefaultQuartzScheduler_Worker-35) CommandAsyncTask::EndActionIfNecessary: All tasks of command ecc942ac-98b9-4e7a-9ff7-ac1229ca1a7c has ended -> executing endAction >2014-08-07 17:56:41,194 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (DefaultQuartzScheduler_Worker-35) CommandAsyncTask::endAction: Ending action for 1 tasks (command ID: ecc942ac-98b9-4e7a-9ff7-ac1229ca1a7c): calling endAction . >2014-08-07 17:56:41,197 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-8-thread-3) CommandAsyncTask::EndCommandAction [within thread] context: Attempting to endAction AddVmTemplate, executionIndex: 0 >2014-08-07 17:56:41,289 INFO [org.ovirt.engine.core.bll.AddVmTemplateCommand] (org.ovirt.thread.pool-8-thread-3) [12896a2a] Ending command successfully: org.ovirt.engine.core.bll.AddVmTemplateCommand >2014-08-07 17:56:41,304 INFO [org.ovirt.engine.core.bll.CreateImageTemplateCommand] (org.ovirt.thread.pool-8-thread-3) [63c518d5] Ending command successfully: org.ovirt.engine.core.bll.CreateImageTemplateCommand >2014-08-07 17:56:41,316 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetImageInfoVDSCommand] (org.ovirt.thread.pool-8-thread-3) [63c518d5] START, GetImageInfoVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, storageDomainId = 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752, imageGroupId = a5dac258-8f4f-48ae-b8a0-1e58a6b190db, imageId = 46253726-8595-4414-b570-faf47fd84c4f), log id: 563ce49e >2014-08-07 17:56:41,339 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetImageInfoVDSCommand] (org.ovirt.thread.pool-8-thread-3) [63c518d5] FINISH, GetImageInfoVDSCommand, return: org.ovirt.engine.core.common.businessentities.DiskImage@2455c18, log id: 563ce49e >2014-08-07 17:56:41,358 INFO [org.ovirt.engine.core.vdsbroker.SetVmStatusVDSCommand] (org.ovirt.thread.pool-8-thread-3) START, SetVmStatusVDSCommand( vmId = 9042bb21-3f27-48bb-ab26-c0d8b18c840d, status = Down, exit status = Normal), log id: 5376710b >2014-08-07 17:56:41,364 INFO [org.ovirt.engine.core.vdsbroker.SetVmStatusVDSCommand] (org.ovirt.thread.pool-8-thread-3) FINISH, SetVmStatusVDSCommand, log id: 5376710b >2014-08-07 17:56:41,393 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-3) Correlation ID: 12896a2a, Call Stack: null, Custom Event ID: -1, Message: Creation of Template blah from VM ovirtnode has been completed. >2014-08-07 17:56:41,394 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-8-thread-3) CommandAsyncTask::HandleEndActionResult [within thread]: endAction for action type AddVmTemplate completed, handling the result. >2014-08-07 17:56:41,395 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-8-thread-3) CommandAsyncTask::HandleEndActionResult [within thread]: endAction for action type AddVmTemplate succeeded, clearing tasks. >2014-08-07 17:56:41,400 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (org.ovirt.thread.pool-8-thread-3) SPMAsyncTask::ClearAsyncTask: Attempting to clear task 0e65f6c1-4537-402b-b37f-abecd8640972 >2014-08-07 17:56:41,402 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-3) START, SPMClearTaskVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, taskId = 0e65f6c1-4537-402b-b37f-abecd8640972), log id: 304f38e2 >2014-08-07 17:56:41,405 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-3) START, HSMClearTaskVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, taskId=0e65f6c1-4537-402b-b37f-abecd8640972), log id: 70c2d12b >2014-08-07 17:56:41,492 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-3) FINISH, HSMClearTaskVDSCommand, log id: 70c2d12b >2014-08-07 17:56:41,493 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-3) FINISH, SPMClearTaskVDSCommand, log id: 304f38e2 >2014-08-07 17:56:41,509 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (org.ovirt.thread.pool-8-thread-3) BaseAsyncTask::RemoveTaskFromDB: Removed task 0e65f6c1-4537-402b-b37f-abecd8640972 from DataBase >2014-08-07 17:56:41,510 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-8-thread-3) CommandAsyncTask::HandleEndActionResult [within thread]: Removing CommandMultiAsyncTasks object for entity ecc942ac-98b9-4e7a-9ff7-ac1229ca1a7c >2014-08-07 17:57:52,688 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (DefaultQuartzScheduler_Worker-40) Setting new tasks map. The map contains now 0 tasks >2014-08-07 17:57:52,690 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (DefaultQuartzScheduler_Worker-40) Cleared all tasks of pool 00000002-0002-0002-0002-0000000001d9. >2014-08-07 18:03:52,809 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-10) Running command: LoginUserCommand internal: false. >2014-08-07 18:03:52,816 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-10) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-08-07 18:03:52,865 INFO [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp--127.0.0.1-8702-10) [39a65cfa] Running command: LogoutBySessionCommand internal: false. >2014-08-07 18:03:52,875 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp--127.0.0.1-8702-10) [1b896c21] Running command: LogoutUserCommand internal: false. >2014-08-07 18:03:52,887 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-10) [1b896c21] Correlation ID: 1b896c21, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-08-07 18:03:53,313 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-8) Running command: LoginUserCommand internal: false. >2014-08-07 18:03:53,385 INFO [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp--127.0.0.1-8702-8) [2c4560d5] Running command: LogoutBySessionCommand internal: false. >2014-08-07 18:03:53,392 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp--127.0.0.1-8702-8) [3a7e065f] Running command: LogoutUserCommand internal: false. >2014-08-07 18:03:53,396 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-8) [3a7e065f] Correlation ID: 3a7e065f, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-08-07 18:03:58,927 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-11) Running command: LoginUserCommand internal: false. >2014-08-07 18:03:58,937 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-11) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-08-07 18:03:59,017 INFO [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp--127.0.0.1-8702-11) [5ff7bf3d] Running command: LogoutBySessionCommand internal: false. >2014-08-07 18:03:59,026 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp--127.0.0.1-8702-11) [600b0838] Running command: LogoutUserCommand internal: false. >2014-08-07 18:03:59,035 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-11) [600b0838] Correlation ID: 600b0838, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-08-07 18:03:59,451 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-4) Running command: LoginUserCommand internal: false. >2014-08-07 18:03:59,469 INFO [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp--127.0.0.1-8702-4) [9e9b557] Running command: LogoutBySessionCommand internal: false. >2014-08-07 18:03:59,479 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp--127.0.0.1-8702-4) [14f1a619] Running command: LogoutUserCommand internal: false. >2014-08-07 18:03:59,488 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-4) [14f1a619] Correlation ID: 14f1a619, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-08-07 18:04:30,853 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-6) Running command: LoginUserCommand internal: false. >2014-08-07 18:04:30,857 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-08-07 18:04:31,754 INFO [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp--127.0.0.1-8702-6) [73365666] Running command: LogoutBySessionCommand internal: false. >2014-08-07 18:04:31,759 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp--127.0.0.1-8702-6) [1467f976] Running command: LogoutUserCommand internal: false. >2014-08-07 18:04:31,762 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) [1467f976] Correlation ID: 1467f976, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-08-07 18:04:54,135 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-9) Running command: LoginUserCommand internal: false. >2014-08-07 18:04:54,140 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-9) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-08-07 18:04:55,034 INFO [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp--127.0.0.1-8702-9) [40f1557b] Running command: LogoutBySessionCommand internal: false. >2014-08-07 18:04:55,038 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp--127.0.0.1-8702-9) [397f7bb0] Running command: LogoutUserCommand internal: false. >2014-08-07 18:04:55,042 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-9) [397f7bb0] Correlation ID: 397f7bb0, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-08-07 18:04:55,470 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-2) Running command: LoginUserCommand internal: false. >2014-08-07 18:04:55,476 INFO [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp--127.0.0.1-8702-2) [71c19599] Running command: LogoutBySessionCommand internal: false. >2014-08-07 18:04:55,488 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp--127.0.0.1-8702-2) [316b6c88] Running command: LogoutUserCommand internal: false. >2014-08-07 18:04:55,499 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-2) [316b6c88] Correlation ID: 316b6c88, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-08-07 18:05:10,985 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-10) Running command: LoginUserCommand internal: false. >2014-08-07 18:05:10,989 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-10) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-08-07 18:05:11,029 INFO [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp--127.0.0.1-8702-10) [b2589ce] Running command: LogoutBySessionCommand internal: false. >2014-08-07 18:05:11,033 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp--127.0.0.1-8702-10) [3fa8c82d] Running command: LogoutUserCommand internal: false. >2014-08-07 18:05:11,039 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-10) [3fa8c82d] Correlation ID: 3fa8c82d, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-08-07 18:05:27,397 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-11) Running command: LoginUserCommand internal: false. >2014-08-07 18:05:27,401 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-11) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-08-07 18:05:27,419 INFO [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp--127.0.0.1-8702-11) [4967df9d] Running command: LogoutBySessionCommand internal: false. >2014-08-07 18:05:27,422 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp--127.0.0.1-8702-11) [1d32d0ed] Running command: LogoutUserCommand internal: false. >2014-08-07 18:05:27,426 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-11) [1d32d0ed] Correlation ID: 1d32d0ed, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-08-07 18:05:39,454 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-1) Running command: LoginUserCommand internal: false. >2014-08-07 18:05:39,457 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-1) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-08-07 18:05:39,466 ERROR [org.ovirt.engine.api.restapi.resource.AbstractBackendResource] (ajp--127.0.0.1-8702-1) Operation Failed: Entity not found: foobar >2014-08-07 18:05:39,509 INFO [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp--127.0.0.1-8702-1) [15f82765] Running command: LogoutBySessionCommand internal: false. >2014-08-07 18:05:39,513 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp--127.0.0.1-8702-1) [533cd946] Running command: LogoutUserCommand internal: false. >2014-08-07 18:05:39,517 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-1) [533cd946] Correlation ID: 533cd946, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-08-07 18:05:53,363 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-5) Running command: LoginUserCommand internal: false. >2014-08-07 18:05:53,367 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-5) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-08-07 18:05:53,372 INFO [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp--127.0.0.1-8702-5) [1b92a7a6] Running command: LogoutBySessionCommand internal: false. >2014-08-07 18:05:53,375 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp--127.0.0.1-8702-5) [48459a6b] Running command: LogoutUserCommand internal: false. >2014-08-07 18:05:53,379 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-5) [48459a6b] Correlation ID: 48459a6b, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-08-07 18:06:13,651 INFO [org.ovirt.engine.core.bll.AddTagCommand] (ajp--127.0.0.1-8702-11) [10b44eca] Running command: AddTagCommand internal: false. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2014-08-07 18:06:13,700 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-11) [10b44eca] Correlation ID: 10b44eca, Call Stack: null, Custom Event ID: -1, Message: New Tag aaa was created by admin. >2014-08-07 18:06:57,998 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-7) Running command: LoginUserCommand internal: false. >2014-08-07 18:06:58,003 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-7) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-08-07 18:06:58,011 INFO [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp--127.0.0.1-8702-7) [58d6d51] Running command: LogoutBySessionCommand internal: false. >2014-08-07 18:06:58,015 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp--127.0.0.1-8702-7) [2acf0930] Running command: LogoutUserCommand internal: false. >2014-08-07 18:06:58,021 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-7) [2acf0930] Correlation ID: 2acf0930, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-08-07 18:07:03,741 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-6) Running command: LoginUserCommand internal: false. >2014-08-07 18:07:03,746 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-08-07 18:07:03,782 INFO [org.ovirt.engine.core.bll.AttachTemplatesToTagCommand] (ajp--127.0.0.1-8702-6) [4c1f7496] Running command: AttachTemplatesToTagCommand internal: false. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2014-08-07 18:07:03,828 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) [4c1f7496] Correlation ID: 4c1f7496, Call Stack: null, Custom Event ID: -1, Message: Tag aaa was attached to Templates(s) blah by admin. >2014-08-07 18:07:04,523 INFO [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp--127.0.0.1-8702-6) [3e1c19c8] Running command: LogoutBySessionCommand internal: false. >2014-08-07 18:07:04,525 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp--127.0.0.1-8702-6) [31d6b45e] Running command: LogoutUserCommand internal: false. >2014-08-07 18:07:04,529 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) [31d6b45e] Correlation ID: 31d6b45e, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-08-07 18:07:26,772 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-4) Running command: LoginUserCommand internal: false. >2014-08-07 18:07:26,776 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-4) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-08-07 18:07:27,593 INFO [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp--127.0.0.1-8702-4) [4ac25a62] Running command: LogoutBySessionCommand internal: false. >2014-08-07 18:07:27,596 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp--127.0.0.1-8702-4) [7a6d7981] Running command: LogoutUserCommand internal: false. >2014-08-07 18:07:27,601 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-4) [7a6d7981] Correlation ID: 7a6d7981, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-08-07 18:07:40,932 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-5) Running command: LoginUserCommand internal: false. >2014-08-07 18:07:40,942 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-5) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-08-07 18:07:41,694 INFO [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp--127.0.0.1-8702-5) [ac1d015] Running command: LogoutBySessionCommand internal: false. >2014-08-07 18:07:41,698 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp--127.0.0.1-8702-5) [23ddc50c] Running command: LogoutUserCommand internal: false. >2014-08-07 18:07:41,704 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-5) [23ddc50c] Correlation ID: 23ddc50c, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-08-07 18:08:17,198 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-6) Running command: LoginUserCommand internal: false. >2014-08-07 18:08:17,202 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-08-07 18:09:28,422 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-11) Running command: LoginUserCommand internal: false. >2014-08-07 18:09:28,427 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-11) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-08-07 18:09:28,439 INFO [org.ovirt.engine.core.bll.AddTagCommand] (ajp--127.0.0.1-8702-11) [7d6aea68] Running command: AddTagCommand internal: false. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2014-08-07 18:09:28,464 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-11) [7d6aea68] Correlation ID: 7d6aea68, Call Stack: null, Custom Event ID: -1, Message: New Tag sracka was created by admin. >2014-08-07 18:09:28,471 INFO [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp--127.0.0.1-8702-11) [10f509a2] Running command: LogoutBySessionCommand internal: false. >2014-08-07 18:09:28,477 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp--127.0.0.1-8702-11) [5e33aeb] Running command: LogoutUserCommand internal: false. >2014-08-07 18:09:28,484 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-11) [5e33aeb] Correlation ID: 5e33aeb, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-08-07 18:09:37,195 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-12) Running command: LoginUserCommand internal: false. >2014-08-07 18:09:37,199 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-12) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-08-07 18:09:37,219 INFO [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp--127.0.0.1-8702-12) [a08f40d] Running command: LogoutBySessionCommand internal: false. >2014-08-07 18:09:37,227 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp--127.0.0.1-8702-12) [1a6fbf5] Running command: LogoutUserCommand internal: false. >2014-08-07 18:09:37,240 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-12) [1a6fbf5] Correlation ID: 1a6fbf5, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-08-07 18:09:54,712 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-3) Running command: LoginUserCommand internal: false. >2014-08-07 18:09:54,717 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-3) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-08-07 18:09:54,733 INFO [org.ovirt.engine.core.bll.AttachTemplatesToTagCommand] (ajp--127.0.0.1-8702-3) [7d689210] Running command: AttachTemplatesToTagCommand internal: false. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2014-08-07 18:09:54,747 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-3) [7d689210] Correlation ID: 7d689210, Call Stack: null, Custom Event ID: -1, Message: Tag sracka was attached to Templates(s) blah by admin. >2014-08-07 18:09:54,754 INFO [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp--127.0.0.1-8702-3) [12dc978e] Running command: LogoutBySessionCommand internal: false. >2014-08-07 18:09:54,757 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp--127.0.0.1-8702-3) [10d9cc95] Running command: LogoutUserCommand internal: false. >2014-08-07 18:09:54,762 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-3) [10d9cc95] Correlation ID: 10d9cc95, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-08-07 18:10:02,610 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-8) Running command: LoginUserCommand internal: false. >2014-08-07 18:10:02,613 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-8) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-08-07 18:10:02,632 INFO [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp--127.0.0.1-8702-8) [2027d0e9] Running command: LogoutBySessionCommand internal: false. >2014-08-07 18:10:02,641 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp--127.0.0.1-8702-8) [4389c5d3] Running command: LogoutUserCommand internal: false. >2014-08-07 18:10:02,644 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-8) [4389c5d3] Correlation ID: 4389c5d3, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-08-07 18:10:17,237 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-4) Running command: LoginUserCommand internal: false. >2014-08-07 18:10:17,248 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-4) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-08-07 18:10:17,627 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-5) Running command: LoginUserCommand internal: false. >2014-08-07 18:10:17,633 INFO [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp--127.0.0.1-8702-5) [6a933c5d] Running command: LogoutBySessionCommand internal: false. >2014-08-07 18:10:17,638 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp--127.0.0.1-8702-5) [70bfc990] Running command: LogoutUserCommand internal: false. >2014-08-07 18:10:17,644 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-5) [70bfc990] Correlation ID: 70bfc990, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-08-07 18:10:36,145 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-9) Running command: LoginUserCommand internal: false. >2014-08-07 18:10:36,149 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-9) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-08-07 18:10:36,159 INFO [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp--127.0.0.1-8702-9) [453ac2dd] Running command: LogoutBySessionCommand internal: false. >2014-08-07 18:10:36,165 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp--127.0.0.1-8702-9) [13dc328] Running command: LogoutUserCommand internal: false. >2014-08-07 18:10:36,170 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-9) [13dc328] Correlation ID: 13dc328, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-08-07 18:10:43,121 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-10) Running command: LoginUserCommand internal: false. >2014-08-07 18:10:43,125 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-10) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-08-07 18:10:43,151 INFO [org.ovirt.engine.core.bll.DetachTemplateFromTagCommand] (ajp--127.0.0.1-8702-10) [604241ea] Running command: DetachTemplateFromTagCommand internal: false. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2014-08-07 18:10:43,174 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-10) [604241ea] Correlation ID: 604241ea, Call Stack: null, Custom Event ID: -1, Message: Tag aaa was detached from Template(s) blah by admin. >2014-08-07 18:10:43,179 INFO [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp--127.0.0.1-8702-10) [4a4be5e] Running command: LogoutBySessionCommand internal: false. >2014-08-07 18:10:43,184 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp--127.0.0.1-8702-10) [38a6afda] Running command: LogoutUserCommand internal: false. >2014-08-07 18:10:43,189 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-10) [38a6afda] Correlation ID: 38a6afda, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-08-07 18:11:01,080 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-11) Running command: LoginUserCommand internal: false. >2014-08-07 18:11:01,093 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-11) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-08-07 18:11:01,111 INFO [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp--127.0.0.1-8702-11) [7c793aad] Running command: LogoutBySessionCommand internal: false. >2014-08-07 18:11:01,115 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp--127.0.0.1-8702-11) [7e118e5c] Running command: LogoutUserCommand internal: false. >2014-08-07 18:11:01,121 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-11) [7e118e5c] Correlation ID: 7e118e5c, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-08-07 18:11:04,478 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-12) Running command: LoginUserCommand internal: false. >2014-08-07 18:11:04,502 INFO [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp--127.0.0.1-8702-12) [7b79c73] Running command: LogoutBySessionCommand internal: false. >2014-08-07 18:11:04,505 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp--127.0.0.1-8702-12) [1c4fbc26] Running command: LogoutUserCommand internal: false. >2014-08-07 18:11:04,509 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-12) [1c4fbc26] Correlation ID: 1c4fbc26, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-08-07 18:11:18,531 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-2) Running command: LoginUserCommand internal: false. >2014-08-07 18:11:18,537 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-2) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-08-07 18:11:18,544 INFO [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp--127.0.0.1-8702-2) [60fc0d21] Running command: LogoutBySessionCommand internal: false. >2014-08-07 18:11:18,547 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp--127.0.0.1-8702-2) [30286889] Running command: LogoutUserCommand internal: false. >2014-08-07 18:11:18,551 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-2) [30286889] Correlation ID: 30286889, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-08-07 18:11:18,886 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-4) Running command: LoginUserCommand internal: false. >2014-08-07 18:11:18,892 INFO [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp--127.0.0.1-8702-4) [39a55340] Running command: LogoutBySessionCommand internal: false. >2014-08-07 18:11:18,895 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp--127.0.0.1-8702-4) [7aa1feb0] Running command: LogoutUserCommand internal: false. >2014-08-07 18:11:18,898 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-4) [7aa1feb0] Correlation ID: 7aa1feb0, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-08-07 18:20:41,078 WARN [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-8-thread-10) Executing a command: java.util.concurrent.FutureTask , but note that there are 1 tasks in the queue. >2014-08-07 18:23:48,978 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ListVDSCommand] (DefaultQuartzScheduler_Worker-77) Command ListVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-08-07 18:23:48,980 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-77) Host rack is not responding. It will stay in Connecting state for a grace period of $80 seconds and after that an attempt to fence the host will be issued. >2014-08-07 18:23:48,983 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-77) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:407) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.fetchRunningVms(VdsUpdateRunTimeInfo.java:971) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVmStats(VdsUpdateRunTimeInfo.java:920) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:638) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor21.invoke(Unknown Source) [:1.7.0_55] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.ConnectException: Connection refused > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_55] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_55] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_55] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_55] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_55] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_55] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_55] > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] > at com.sun.proxy.$Proxy125.list(Unknown Source) > at sun.reflect.GeneratedMethodAccessor29.invoke(Unknown Source) [:1.7.0_55] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55] > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:242) [vdsbroker.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_55] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_55] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_55] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_55] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_55] > >2014-08-07 18:23:49,024 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-77) Failed to refresh VDS , vds = 17f7fe56-5f32-40ef-b098-611b84484e0b : rack, VDS Network Error, continuing. >java.net.ConnectException: Connection refused >2014-08-07 18:23:52,403 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-84) Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 44559ec0-6bef-491c-bbd6-e67278f37987 status from Active to Unknown, reason : no reporting hosts >2014-08-07 18:23:52,407 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-84) Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Active to Unknown, reason : no reporting hosts >2014-08-07 18:23:52,411 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-84) Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 63fe3516-4170-402a-9dc8-6d3eab1c7933 status from Active to Unknown, reason : no reporting hosts >2014-08-07 18:23:52,497 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-84) [7f085c60] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-08-07 18:23:52,522 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-66) START, GetHardwareInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]), log id: 23690360 >2014-08-07 18:23:52,525 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-84) [7f085c60] Correlation ID: 7f085c60, Call Stack: null, Custom Event ID: -1, Message: Invalid status on Data Center Default. Setting status to Non Responsive. >2014-08-07 18:23:52,544 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-84) [7f085c60] Command org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand return value > >TaskStatusListReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=654, mMessage=Not SPM: ()]] > >2014-08-07 18:23:52,548 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-84) [7f085c60] HostName = rack >2014-08-07 18:23:52,550 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-84) [7f085c60] Command HSMGetAllTasksStatusesVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b) execution failed. Exception: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM: () >2014-08-07 18:23:52,694 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-66) FINISH, GetHardwareInfoVDSCommand, log id: 23690360 >2014-08-07 18:23:52,744 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-66) [aad989f] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS >2014-08-07 18:23:52,763 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-66) [22642369] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-08-07 18:23:52,791 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-66) [69dd15bf] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-08-07 18:23:52,817 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-66) [69dd15bf] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, storageType = NFS, connectionList = [{ id: 05047c9a-152e-4ef2-b118-4a1753c1eae0, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35export, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 7fe036a2-2dfc-4565-b936-69a678cf0613, connection: pb-rh35.rhev.lab.eng.brq.redhat.com:/var/lib/exports/iso, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 6c970424 >2014-08-07 18:23:52,849 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-66) [69dd15bf] FINISH, ConnectStorageServerVDSCommand, return: {7fe036a2-2dfc-4565-b936-69a678cf0613=0, 5083f370-af09-457b-9b73-3440e8b65a07=0, 05047c9a-152e-4ef2-b118-4a1753c1eae0=0}, log id: 6c970424 >2014-08-07 18:23:52,851 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-66) [69dd15bf] Host rack storage connection was succeeded >2014-08-07 18:23:52,868 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-3) START, ConnectStoragePoolVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, masterVersion = 1), log id: 61f06e9 >2014-08-07 18:23:53,354 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-3) FINISH, ConnectStoragePoolVDSCommand, log id: 61f06e9 >2014-08-07 18:23:53,369 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-66) [69dd15bf] START, SetMOMPolicyParametersVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 1d18d97b >2014-08-07 18:23:53,461 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-66) [69dd15bf] FINISH, SetMOMPolicyParametersVDSCommand, log id: 1d18d97b >2014-08-07 18:23:53,469 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-66) [69dd15bf] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host rack was set to Up. >2014-08-07 18:23:53,476 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-66) [69dd15bf] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host rack. >2014-08-07 18:23:53,501 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-66) [215ad4c4] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS >2014-08-07 18:24:02,750 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-99) hostFromVds::selectedVds - rack, spmStatus Free, storage pool Default >2014-08-07 18:24:02,766 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-99) starting spm on vds rack, storage pool Default, prevId -1, LVER -1 >2014-08-07 18:24:02,771 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-99) START, SpmStartVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, prevId=-1, prevLVER=-1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 42cf5663 >2014-08-07 18:24:02,820 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-99) spmStart polling started: taskId = 4d485da8-e4b0-4ec2-97f9-cf7eac7579a4 >2014-08-07 18:24:03,829 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-99) spmStart polling ended: taskId = 4d485da8-e4b0-4ec2-97f9-cf7eac7579a4 task status = finished >2014-08-07 18:24:03,885 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-99) spmStart polling ended, spm status: SPM >2014-08-07 18:24:03,888 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-99) START, HSMClearTaskVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, taskId=4d485da8-e4b0-4ec2-97f9-cf7eac7579a4), log id: 11235c5d >2014-08-07 18:24:03,934 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-99) FINISH, HSMClearTaskVDSCommand, log id: 11235c5d >2014-08-07 18:24:03,935 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-99) FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@65e9d366, log id: 42cf5663 >2014-08-07 18:24:03,938 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-99) Initialize Irs proxy from vds: 10.34.62.208 >2014-08-07 18:24:03,944 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-99) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208). >2014-08-07 18:24:03,958 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-19) START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 17655a77 >2014-08-07 18:24:04,004 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-19) -- executeIrsBrokerCommand: Attempting on storage pool 00000002-0002-0002-0002-0000000001d9 >2014-08-07 18:24:04,007 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-19) START, HSMGetAllTasksInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 14e821e3 >2014-08-07 18:24:04,012 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-19) FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 14e821e3 >2014-08-07 18:24:04,013 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-19) FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 17655a77 >2014-08-07 18:24:04,014 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (org.ovirt.thread.pool-8-thread-19) Discovered no tasks on Storage Pool Default >2014-08-07 18:37:52,684 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-21) Principal admin::internal synchronized >2014-08-07 18:37:53,681 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-15) Attempting to update VMs/Templates Ovf. >2014-08-07 18:37:53,815 INFO [org.ovirt.engine.core.bll.ProcessOvfUpdateForStorageDomainCommand] (DefaultQuartzScheduler_Worker-15) [449fe956] Lock Acquired to object EngineLock [exclusiveLocks= key: 07c061c2-77f4-4c02-8402-e4dff773b216 value: DISK >key: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 value: STORAGE >key: 90108fe1-56a4-4f0e-b301-fa0ef46ba717 value: DISK >, sharedLocks= key: 00000002-0002-0002-0002-0000000001d9 value: OVF_UPDATE >] >2014-08-07 18:37:53,820 INFO [org.ovirt.engine.core.bll.ProcessOvfUpdateForStorageDomainCommand] (DefaultQuartzScheduler_Worker-15) [449fe956] Running command: ProcessOvfUpdateForStorageDomainCommand internal: true. Entities affected : ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage >2014-08-07 18:37:53,866 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (DefaultQuartzScheduler_Worker-15) [449fe956] START, SetVolumeDescriptionVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, storageDomainId = 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752, imageGroupId = 90108fe1-56a4-4f0e-b301-fa0ef46ba717, imageId = 8596469e-37df-4d9e-9793-d238238131f5), log id: 2dae23a2 >2014-08-07 18:37:53,868 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (DefaultQuartzScheduler_Worker-15) [449fe956] -- executeIrsBrokerCommand: calling 'setVolumeDescription' >2014-08-07 18:37:53,869 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (DefaultQuartzScheduler_Worker-15) [449fe956] -- setVolumeDescription parameters: > spUUID=00000002-0002-0002-0002-0000000001d9 > sdUUID=71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 > imageGroupGUID=90108fe1-56a4-4f0e-b301-fa0ef46ba717 > volUUID=8596469e-37df-4d9e-9793-d238238131f5 > description={"Updated":false,"Disk Description":"OVF_STORE","Storage Domains":[{"uuid":"71f82fcd-4fa6-4f56-b228-f4fa4bd4f752"}],"Last Updated":"Wed Jul 30 11:24:01 CEST 2014"} > >2014-08-07 18:37:54,002 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (DefaultQuartzScheduler_Worker-15) [449fe956] FINISH, SetVolumeDescriptionVDSCommand, log id: 2dae23a2 >2014-08-07 18:37:54,076 INFO [org.ovirt.engine.core.bll.UploadStreamCommand] (DefaultQuartzScheduler_Worker-15) [5cf4c9c9] Running command: UploadStreamCommand internal: true. Entities affected : ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage >2014-08-07 18:37:54,085 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.UploadStreamVDSCommand] (DefaultQuartzScheduler_Worker-15) [5cf4c9c9] START, UploadStreamVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 5e622a41 >2014-08-07 18:37:54,086 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.UploadStreamVDSCommand] (DefaultQuartzScheduler_Worker-15) [5cf4c9c9] -- executeVdsBrokerCommand: >2014-08-07 18:37:54,088 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.UploadStreamVDSCommand] (DefaultQuartzScheduler_Worker-15) [5cf4c9c9] -- parameters: > spUUID=00000002-0002-0002-0002-0000000001d9 > sdUUID=71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 > imageGUID=90108fe1-56a4-4f0e-b301-fa0ef46ba717 > volUUID=8596469e-37df-4d9e-9793-d238238131f5 > size=20,480 > >2014-08-07 18:37:54,710 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.UploadStreamVDSCommand] (DefaultQuartzScheduler_Worker-15) [5cf4c9c9] FINISH, UploadStreamVDSCommand, log id: 5e622a41 >2014-08-07 18:37:54,732 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (DefaultQuartzScheduler_Worker-15) [5cf4c9c9] CommandAsyncTask::Adding CommandMultiAsyncTasks object for command 29253b22-c73a-4328-a4e2-43d6756e8cfa >2014-08-07 18:37:54,733 INFO [org.ovirt.engine.core.bll.CommandMultiAsyncTasks] (DefaultQuartzScheduler_Worker-15) [5cf4c9c9] CommandMultiAsyncTasks::AttachTask: Attaching task 1cd6a309-b71c-4a1d-9bcf-75e18d6d99f9 to command 29253b22-c73a-4328-a4e2-43d6756e8cfa. >2014-08-07 18:37:54,743 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (DefaultQuartzScheduler_Worker-15) [5cf4c9c9] Adding task 1cd6a309-b71c-4a1d-9bcf-75e18d6d99f9 (Parent Command ProcessOvfUpdateForStorageDomain, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters), polling hasn't started yet.. >2014-08-07 18:37:54,750 INFO [org.ovirt.engine.core.bll.UploadStreamCommand] (DefaultQuartzScheduler_Worker-15) [5cf4c9c9] Lock freed to object EngineLock [exclusiveLocks= key: 07c061c2-77f4-4c02-8402-e4dff773b216 value: DISK >key: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 value: STORAGE >key: 90108fe1-56a4-4f0e-b301-fa0ef46ba717 value: DISK >, sharedLocks= key: 00000002-0002-0002-0002-0000000001d9 value: OVF_UPDATE >] >2014-08-07 18:37:54,763 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (DefaultQuartzScheduler_Worker-15) [5cf4c9c9] START, SetVolumeDescriptionVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, storageDomainId = 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752, imageGroupId = 90108fe1-56a4-4f0e-b301-fa0ef46ba717, imageId = 8596469e-37df-4d9e-9793-d238238131f5), log id: 134d15b3 >2014-08-07 18:37:54,767 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (DefaultQuartzScheduler_Worker-15) [5cf4c9c9] -- executeIrsBrokerCommand: calling 'setVolumeDescription' >2014-08-07 18:37:54,768 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (DefaultQuartzScheduler_Worker-15) [5cf4c9c9] -- setVolumeDescription parameters: > spUUID=00000002-0002-0002-0002-0000000001d9 > sdUUID=71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 > imageGroupGUID=90108fe1-56a4-4f0e-b301-fa0ef46ba717 > volUUID=8596469e-37df-4d9e-9793-d238238131f5 > description={"Updated":true,"Disk Description":"OVF_STORE","Storage Domains":[{"uuid":"71f82fcd-4fa6-4f56-b228-f4fa4bd4f752"}],"Last Updated":"Thu Aug 07 18:37:53 CEST 2014","Size":20480} > >2014-08-07 18:37:54,902 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (DefaultQuartzScheduler_Worker-15) [5cf4c9c9] FINISH, SetVolumeDescriptionVDSCommand, log id: 134d15b3 >2014-08-07 18:37:54,914 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (DefaultQuartzScheduler_Worker-15) [5cf4c9c9] START, SetVolumeDescriptionVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, storageDomainId = 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752, imageGroupId = 07c061c2-77f4-4c02-8402-e4dff773b216, imageId = 1f5f82d1-85c2-4386-8764-02bab6dcbc21), log id: 36132e09 >2014-08-07 18:37:54,923 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (DefaultQuartzScheduler_Worker-15) [5cf4c9c9] -- executeIrsBrokerCommand: calling 'setVolumeDescription' >2014-08-07 18:37:54,929 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (DefaultQuartzScheduler_Worker-15) [5cf4c9c9] -- setVolumeDescription parameters: > spUUID=00000002-0002-0002-0002-0000000001d9 > sdUUID=71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 > imageGroupGUID=07c061c2-77f4-4c02-8402-e4dff773b216 > volUUID=1f5f82d1-85c2-4386-8764-02bab6dcbc21 > description={"Updated":false,"Disk Description":"OVF_STORE","Storage Domains":[{"uuid":"71f82fcd-4fa6-4f56-b228-f4fa4bd4f752"}],"Last Updated":"Wed Jul 30 11:24:01 CEST 2014"} > >2014-08-07 18:37:55,073 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (DefaultQuartzScheduler_Worker-15) [5cf4c9c9] FINISH, SetVolumeDescriptionVDSCommand, log id: 36132e09 >2014-08-07 18:37:55,099 INFO [org.ovirt.engine.core.bll.UploadStreamCommand] (DefaultQuartzScheduler_Worker-15) [6fcc1be5] Running command: UploadStreamCommand internal: true. Entities affected : ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage >2014-08-07 18:37:55,103 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.UploadStreamVDSCommand] (DefaultQuartzScheduler_Worker-15) [6fcc1be5] START, UploadStreamVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: b6c7c5d >2014-08-07 18:37:55,106 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.UploadStreamVDSCommand] (DefaultQuartzScheduler_Worker-15) [6fcc1be5] -- executeVdsBrokerCommand: >2014-08-07 18:37:55,107 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.UploadStreamVDSCommand] (DefaultQuartzScheduler_Worker-15) [6fcc1be5] -- parameters: > spUUID=00000002-0002-0002-0002-0000000001d9 > sdUUID=71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 > imageGUID=07c061c2-77f4-4c02-8402-e4dff773b216 > volUUID=1f5f82d1-85c2-4386-8764-02bab6dcbc21 > size=20,480 > >2014-08-07 18:37:55,763 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.UploadStreamVDSCommand] (DefaultQuartzScheduler_Worker-15) [6fcc1be5] FINISH, UploadStreamVDSCommand, log id: b6c7c5d >2014-08-07 18:37:55,766 INFO [org.ovirt.engine.core.bll.CommandMultiAsyncTasks] (DefaultQuartzScheduler_Worker-15) [6fcc1be5] CommandMultiAsyncTasks::AttachTask: Attaching task 982f8689-dddf-4b8a-bd50-d1f2c79be801 to command 29253b22-c73a-4328-a4e2-43d6756e8cfa. >2014-08-07 18:37:55,799 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (DefaultQuartzScheduler_Worker-15) [6fcc1be5] Adding task 982f8689-dddf-4b8a-bd50-d1f2c79be801 (Parent Command ProcessOvfUpdateForStorageDomain, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters), polling hasn't started yet.. >2014-08-07 18:37:55,808 WARN [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (DefaultQuartzScheduler_Worker-15) [6fcc1be5] Trying to release a shared lock for key: 00000002-0002-0002-0002-0000000001d9OVF_UPDATE , but lock does not exist >2014-08-07 18:37:55,809 WARN [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (DefaultQuartzScheduler_Worker-15) [6fcc1be5] Trying to release exclusive lock which does not exist, lock key: 07c061c2-77f4-4c02-8402-e4dff773b216DISK >2014-08-07 18:37:55,811 WARN [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (DefaultQuartzScheduler_Worker-15) [6fcc1be5] Trying to release exclusive lock which does not exist, lock key: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752STORAGE >2014-08-07 18:37:55,812 WARN [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (DefaultQuartzScheduler_Worker-15) [6fcc1be5] Trying to release exclusive lock which does not exist, lock key: 90108fe1-56a4-4f0e-b301-fa0ef46ba717DISK >2014-08-07 18:37:55,820 INFO [org.ovirt.engine.core.bll.UploadStreamCommand] (DefaultQuartzScheduler_Worker-15) [6fcc1be5] Lock freed to object EngineLock [exclusiveLocks= key: 07c061c2-77f4-4c02-8402-e4dff773b216 value: DISK >key: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 value: STORAGE >key: 90108fe1-56a4-4f0e-b301-fa0ef46ba717 value: DISK >, sharedLocks= key: 00000002-0002-0002-0002-0000000001d9 value: OVF_UPDATE >] >2014-08-07 18:37:55,830 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (DefaultQuartzScheduler_Worker-15) [6fcc1be5] START, SetVolumeDescriptionVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, storageDomainId = 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752, imageGroupId = 07c061c2-77f4-4c02-8402-e4dff773b216, imageId = 1f5f82d1-85c2-4386-8764-02bab6dcbc21), log id: 6ca80435 >2014-08-07 18:37:55,833 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (DefaultQuartzScheduler_Worker-15) [6fcc1be5] -- executeIrsBrokerCommand: calling 'setVolumeDescription' >2014-08-07 18:37:55,835 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (DefaultQuartzScheduler_Worker-15) [6fcc1be5] -- setVolumeDescription parameters: > spUUID=00000002-0002-0002-0002-0000000001d9 > sdUUID=71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 > imageGroupGUID=07c061c2-77f4-4c02-8402-e4dff773b216 > volUUID=1f5f82d1-85c2-4386-8764-02bab6dcbc21 > description={"Updated":true,"Disk Description":"OVF_STORE","Storage Domains":[{"uuid":"71f82fcd-4fa6-4f56-b228-f4fa4bd4f752"}],"Last Updated":"Thu Aug 07 18:37:53 CEST 2014","Size":20480} > >2014-08-07 18:37:55,968 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (DefaultQuartzScheduler_Worker-15) [6fcc1be5] FINISH, SetVolumeDescriptionVDSCommand, log id: 6ca80435 >2014-08-07 18:37:55,972 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (DefaultQuartzScheduler_Worker-15) [6fcc1be5] BaseAsyncTask::startPollingTask: Starting to poll task 1cd6a309-b71c-4a1d-9bcf-75e18d6d99f9. >2014-08-07 18:37:55,973 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (DefaultQuartzScheduler_Worker-15) [6fcc1be5] BaseAsyncTask::startPollingTask: Starting to poll task 982f8689-dddf-4b8a-bd50-d1f2c79be801. >2014-08-07 18:38:01,290 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (DefaultQuartzScheduler_Worker-55) Polling and updating Async Tasks: 2 tasks, 2 tasks to poll now >2014-08-07 18:38:01,296 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (DefaultQuartzScheduler_Worker-55) SPMAsyncTask::PollTask: Polling task 982f8689-dddf-4b8a-bd50-d1f2c79be801 (Parent Command ProcessOvfUpdateForStorageDomain, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) returned status finished, result 'success'. >2014-08-07 18:38:01,299 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (DefaultQuartzScheduler_Worker-55) BaseAsyncTask::OnTaskEndSuccess: Task 982f8689-dddf-4b8a-bd50-d1f2c79be801 (Parent Command ProcessOvfUpdateForStorageDomain, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) ended successfully. >2014-08-07 18:38:01,301 INFO [org.ovirt.engine.core.bll.CommandMultiAsyncTasks] (DefaultQuartzScheduler_Worker-55) Task with DB Task ID 2134bb45-6102-4fe9-9913-ddb0137cb290 and VDSM Task ID 1cd6a309-b71c-4a1d-9bcf-75e18d6d99f9 is in state Polling. End action for command 29253b22-c73a-4328-a4e2-43d6756e8cfa will proceed when all the entitys tasks are completed. >2014-08-07 18:38:01,303 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (DefaultQuartzScheduler_Worker-55) SPMAsyncTask::PollTask: Polling task 1cd6a309-b71c-4a1d-9bcf-75e18d6d99f9 (Parent Command ProcessOvfUpdateForStorageDomain, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) returned status finished, result 'success'. >2014-08-07 18:38:01,304 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (DefaultQuartzScheduler_Worker-55) BaseAsyncTask::OnTaskEndSuccess: Task 1cd6a309-b71c-4a1d-9bcf-75e18d6d99f9 (Parent Command ProcessOvfUpdateForStorageDomain, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) ended successfully. >2014-08-07 18:38:01,305 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (DefaultQuartzScheduler_Worker-55) CommandAsyncTask::EndActionIfNecessary: All tasks of command 29253b22-c73a-4328-a4e2-43d6756e8cfa has ended -> executing endAction >2014-08-07 18:38:01,306 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (DefaultQuartzScheduler_Worker-55) CommandAsyncTask::endAction: Ending action for 2 tasks (command ID: 29253b22-c73a-4328-a4e2-43d6756e8cfa): calling endAction . >2014-08-07 18:38:01,308 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-8-thread-37) CommandAsyncTask::EndCommandAction [within thread] context: Attempting to endAction ProcessOvfUpdateForStorageDomain, executionIndex: 0 >2014-08-07 18:38:01,345 INFO [org.ovirt.engine.core.bll.ProcessOvfUpdateForStorageDomainCommand] (org.ovirt.thread.pool-8-thread-37) [449fe956] Ending command successfully: org.ovirt.engine.core.bll.ProcessOvfUpdateForStorageDomainCommand >2014-08-07 18:38:01,346 WARN [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (org.ovirt.thread.pool-8-thread-37) [449fe956] Trying to release a shared lock for key: 00000002-0002-0002-0002-0000000001d9OVF_UPDATE , but lock does not exist >2014-08-07 18:38:01,348 WARN [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (org.ovirt.thread.pool-8-thread-37) [449fe956] Trying to release exclusive lock which does not exist, lock key: 07c061c2-77f4-4c02-8402-e4dff773b216DISK >2014-08-07 18:38:01,349 WARN [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (org.ovirt.thread.pool-8-thread-37) [449fe956] Trying to release exclusive lock which does not exist, lock key: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752STORAGE >2014-08-07 18:38:01,350 WARN [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (org.ovirt.thread.pool-8-thread-37) [449fe956] Trying to release exclusive lock which does not exist, lock key: 90108fe1-56a4-4f0e-b301-fa0ef46ba717DISK >2014-08-07 18:38:01,352 INFO [org.ovirt.engine.core.bll.ProcessOvfUpdateForStorageDomainCommand] (org.ovirt.thread.pool-8-thread-37) [449fe956] Lock freed to object EngineLock [exclusiveLocks= key: 07c061c2-77f4-4c02-8402-e4dff773b216 value: DISK >key: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 value: STORAGE >key: 90108fe1-56a4-4f0e-b301-fa0ef46ba717 value: DISK >, sharedLocks= key: 00000002-0002-0002-0002-0000000001d9 value: OVF_UPDATE >] >2014-08-07 18:38:01,353 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-8-thread-37) [449fe956] CommandAsyncTask::HandleEndActionResult [within thread]: endAction for action type ProcessOvfUpdateForStorageDomain completed, handling the result. >2014-08-07 18:38:01,355 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-8-thread-37) [449fe956] CommandAsyncTask::HandleEndActionResult [within thread]: endAction for action type ProcessOvfUpdateForStorageDomain succeeded, clearing tasks. >2014-08-07 18:38:01,356 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (org.ovirt.thread.pool-8-thread-37) [449fe956] SPMAsyncTask::ClearAsyncTask: Attempting to clear task 982f8689-dddf-4b8a-bd50-d1f2c79be801 >2014-08-07 18:38:01,357 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-37) [449fe956] START, SPMClearTaskVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, taskId = 982f8689-dddf-4b8a-bd50-d1f2c79be801), log id: 7bc1f07e >2014-08-07 18:38:01,363 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-37) [449fe956] START, HSMClearTaskVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, taskId=982f8689-dddf-4b8a-bd50-d1f2c79be801), log id: 6bab51e8 >2014-08-07 18:38:01,414 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-37) [449fe956] FINISH, HSMClearTaskVDSCommand, log id: 6bab51e8 >2014-08-07 18:38:01,415 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-37) [449fe956] FINISH, SPMClearTaskVDSCommand, log id: 7bc1f07e >2014-08-07 18:38:01,422 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (org.ovirt.thread.pool-8-thread-37) [449fe956] BaseAsyncTask::RemoveTaskFromDB: Removed task 982f8689-dddf-4b8a-bd50-d1f2c79be801 from DataBase >2014-08-07 18:38:01,423 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (org.ovirt.thread.pool-8-thread-37) [449fe956] SPMAsyncTask::ClearAsyncTask: Attempting to clear task 1cd6a309-b71c-4a1d-9bcf-75e18d6d99f9 >2014-08-07 18:38:01,424 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-37) [449fe956] START, SPMClearTaskVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, taskId = 1cd6a309-b71c-4a1d-9bcf-75e18d6d99f9), log id: 183344c8 >2014-08-07 18:38:01,427 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-37) [449fe956] START, HSMClearTaskVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, taskId=1cd6a309-b71c-4a1d-9bcf-75e18d6d99f9), log id: 4de8b2e1 >2014-08-07 18:38:01,496 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-37) [449fe956] FINISH, HSMClearTaskVDSCommand, log id: 4de8b2e1 >2014-08-07 18:38:01,497 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-37) [449fe956] FINISH, SPMClearTaskVDSCommand, log id: 183344c8 >2014-08-07 18:38:01,502 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (org.ovirt.thread.pool-8-thread-37) [449fe956] BaseAsyncTask::RemoveTaskFromDB: Removed task 1cd6a309-b71c-4a1d-9bcf-75e18d6d99f9 from DataBase >2014-08-07 18:38:01,503 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-8-thread-37) [449fe956] CommandAsyncTask::HandleEndActionResult [within thread]: Removing CommandMultiAsyncTasks object for entity 29253b22-c73a-4328-a4e2-43d6756e8cfa >2014-08-07 18:39:22,725 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (DefaultQuartzScheduler_Worker-49) Setting new tasks map. The map contains now 0 tasks >2014-08-07 18:39:22,726 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (DefaultQuartzScheduler_Worker-49) Cleared all tasks of pool 00000002-0002-0002-0002-0000000001d9. >2014-08-07 18:42:52,165 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-45) START, GetFileStatsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 21ab8d34 >2014-08-07 18:42:52,181 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-45) FINISH, GetFileStatsVDSCommand, return: {test.iso={status=0, ctime=1407430000.0, size=4}}, log id: 21ab8d34 >2014-08-07 18:42:52,187 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (org.ovirt.thread.pool-8-thread-45) Finished automatic refresh process for ISO file type with success, for storage domain id 63fe3516-4170-402a-9dc8-6d3eab1c7933. >2014-08-07 19:37:52,699 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-98) Principal admin::internal synchronized >2014-08-07 19:37:55,974 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-8) Attempting to update VMs/Templates Ovf. >2014-08-07 19:42:52,200 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-23) START, GetFileStatsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 2e262ca >2014-08-07 19:42:52,223 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-23) FINISH, GetFileStatsVDSCommand, return: {test.iso={status=0, ctime=1407430000.0, size=4}}, log id: 2e262ca >2014-08-07 19:42:52,228 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (org.ovirt.thread.pool-8-thread-23) Finished automatic refresh process for ISO file type with success, for storage domain id 63fe3516-4170-402a-9dc8-6d3eab1c7933. >2014-08-07 20:37:52,705 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-6) [46247d52] Principal admin::internal synchronized >2014-08-07 20:37:55,995 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-97) Attempting to update VMs/Templates Ovf. >2014-08-07 20:42:52,238 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-50) [46247d52] START, GetFileStatsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 36d349dd >2014-08-07 20:42:52,253 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-50) [46247d52] FINISH, GetFileStatsVDSCommand, return: {test.iso={status=0, ctime=1407430000.0, size=4}}, log id: 36d349dd >2014-08-07 20:42:52,259 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (org.ovirt.thread.pool-8-thread-50) [46247d52] Finished automatic refresh process for ISO file type with success, for storage domain id 63fe3516-4170-402a-9dc8-6d3eab1c7933. >2014-08-07 21:37:52,712 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-84) [7f085c60] Principal admin::internal synchronized >2014-08-07 21:37:56,033 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-75) Attempting to update VMs/Templates Ovf. >2014-08-07 21:42:52,273 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-50) START, GetFileStatsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 2e77b3f >2014-08-07 21:42:52,294 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-50) FINISH, GetFileStatsVDSCommand, return: {test.iso={status=0, ctime=1407430000.0, size=4}}, log id: 2e77b3f >2014-08-07 21:42:52,301 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (org.ovirt.thread.pool-8-thread-50) Finished automatic refresh process for ISO file type with success, for storage domain id 63fe3516-4170-402a-9dc8-6d3eab1c7933. >2014-08-07 22:37:52,720 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-96) Principal admin::internal synchronized >2014-08-07 22:37:56,073 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-6) [46247d52] Attempting to update VMs/Templates Ovf. >2014-08-07 22:42:52,312 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-42) START, GetFileStatsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 51ddef45 >2014-08-07 22:42:52,329 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-42) FINISH, GetFileStatsVDSCommand, return: {test.iso={status=0, ctime=1407430000.0, size=4}}, log id: 51ddef45 >2014-08-07 22:42:52,336 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (org.ovirt.thread.pool-8-thread-42) Finished automatic refresh process for ISO file type with success, for storage domain id 63fe3516-4170-402a-9dc8-6d3eab1c7933. >2014-08-07 23:37:52,727 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-17) Principal admin::internal synchronized >2014-08-07 23:37:56,116 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-82) Attempting to update VMs/Templates Ovf. >2014-08-07 23:42:52,344 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-7) START, GetFileStatsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 41f6b132 >2014-08-07 23:42:52,360 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-7) FINISH, GetFileStatsVDSCommand, return: {test.iso={status=0, ctime=1407430000.0, size=4}}, log id: 41f6b132 >2014-08-07 23:42:52,366 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (org.ovirt.thread.pool-8-thread-7) Finished automatic refresh process for ISO file type with success, for storage domain id 63fe3516-4170-402a-9dc8-6d3eab1c7933. >2014-08-08 00:37:52,733 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-62) Principal admin::internal synchronized >2014-08-08 00:37:56,138 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-39) Attempting to update VMs/Templates Ovf. >2014-08-08 00:42:52,375 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-11) START, GetFileStatsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 28c33dd6 >2014-08-08 00:42:52,393 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-11) FINISH, GetFileStatsVDSCommand, return: {test.iso={status=0, ctime=1407430000.0, size=4}}, log id: 28c33dd6 >2014-08-08 00:42:52,398 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (org.ovirt.thread.pool-8-thread-11) Finished automatic refresh process for ISO file type with success, for storage domain id 63fe3516-4170-402a-9dc8-6d3eab1c7933. >2014-08-08 01:37:52,867 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-29) Principal admin::internal synchronized >2014-08-08 01:37:56,156 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-17) Attempting to update VMs/Templates Ovf. >2014-08-08 01:42:52,406 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-15) START, GetFileStatsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: f6c7da7 >2014-08-08 01:42:52,422 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-15) FINISH, GetFileStatsVDSCommand, return: {test.iso={status=0, ctime=1407430000.0, size=4}}, log id: f6c7da7 >2014-08-08 01:42:52,427 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (org.ovirt.thread.pool-8-thread-15) Finished automatic refresh process for ISO file type with success, for storage domain id 63fe3516-4170-402a-9dc8-6d3eab1c7933. >2014-08-08 02:37:52,880 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-1) Principal admin::internal synchronized >2014-08-08 02:37:56,170 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-3) Attempting to update VMs/Templates Ovf. >2014-08-08 02:42:52,437 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-12) START, GetFileStatsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 732fc359 >2014-08-08 02:42:52,454 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-12) FINISH, GetFileStatsVDSCommand, return: {test.iso={status=0, ctime=1407430000.0, size=4}}, log id: 732fc359 >2014-08-08 02:42:52,459 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (org.ovirt.thread.pool-8-thread-12) Finished automatic refresh process for ISO file type with success, for storage domain id 63fe3516-4170-402a-9dc8-6d3eab1c7933. >2014-08-08 03:35:35,002 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-19) [175991ea] Start deleteAgedOutAuditLogs >2014-08-08 03:35:35,004 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-38) Start deleteAgedOutCommandEntities >2014-08-08 03:35:35,024 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-38) Finished deleteAgedOutCommandEntities >2014-08-08 03:35:35,026 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-19) [175991ea] Finished deleteAgedOutAuditLogs >2014-08-08 03:37:52,886 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-93) Principal admin::internal synchronized >2014-08-08 03:37:56,194 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-97) Attempting to update VMs/Templates Ovf. >2014-08-08 03:42:52,469 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-41) START, GetFileStatsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 2f159380 >2014-08-08 03:42:52,485 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-41) FINISH, GetFileStatsVDSCommand, return: {test.iso={status=0, ctime=1407430000.0, size=4}}, log id: 2f159380 >2014-08-08 03:42:52,492 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (org.ovirt.thread.pool-8-thread-41) Finished automatic refresh process for ISO file type with success, for storage domain id 63fe3516-4170-402a-9dc8-6d3eab1c7933. >2014-08-08 04:37:52,894 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-6) [46247d52] Principal admin::internal synchronized >2014-08-08 04:37:56,257 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-76) Attempting to update VMs/Templates Ovf. >2014-08-08 04:42:52,500 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-3) START, GetFileStatsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 7996de58 >2014-08-08 04:42:52,517 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-3) FINISH, GetFileStatsVDSCommand, return: {test.iso={status=0, ctime=1407430000.0, size=4}}, log id: 7996de58 >2014-08-08 04:42:52,522 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (org.ovirt.thread.pool-8-thread-3) Finished automatic refresh process for ISO file type with success, for storage domain id 63fe3516-4170-402a-9dc8-6d3eab1c7933. >2014-08-08 05:37:52,907 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-84) [7f085c60] Principal admin::internal synchronized >2014-08-08 05:37:56,306 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-93) Attempting to update VMs/Templates Ovf. >2014-08-08 05:42:52,533 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-27) START, GetFileStatsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 6397d16 >2014-08-08 05:42:52,550 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-27) FINISH, GetFileStatsVDSCommand, return: {test.iso={status=0, ctime=1407430000.0, size=4}}, log id: 6397d16 >2014-08-08 05:42:52,556 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (org.ovirt.thread.pool-8-thread-27) Finished automatic refresh process for ISO file type with success, for storage domain id 63fe3516-4170-402a-9dc8-6d3eab1c7933. >2014-08-08 06:37:52,913 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-82) Principal admin::internal synchronized >2014-08-08 06:37:56,326 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-87) Attempting to update VMs/Templates Ovf. >2014-08-08 06:42:52,564 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-8) START, GetFileStatsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 64eaa2a3 >2014-08-08 06:42:52,582 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-8) FINISH, GetFileStatsVDSCommand, return: {test.iso={status=0, ctime=1407430000.0, size=4}}, log id: 64eaa2a3 >2014-08-08 06:42:52,587 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (org.ovirt.thread.pool-8-thread-8) Finished automatic refresh process for ISO file type with success, for storage domain id 63fe3516-4170-402a-9dc8-6d3eab1c7933. >2014-08-08 07:37:52,918 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-67) Principal admin::internal synchronized >2014-08-08 07:37:56,348 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-54) Attempting to update VMs/Templates Ovf. >2014-08-08 07:42:52,595 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-27) START, GetFileStatsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 7d7a0681 >2014-08-08 07:42:52,612 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-27) FINISH, GetFileStatsVDSCommand, return: {test.iso={status=0, ctime=1407430000.0, size=4}}, log id: 7d7a0681 >2014-08-08 07:42:52,617 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (org.ovirt.thread.pool-8-thread-27) Finished automatic refresh process for ISO file type with success, for storage domain id 63fe3516-4170-402a-9dc8-6d3eab1c7933. >2014-08-08 08:37:52,924 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-34) Principal admin::internal synchronized >2014-08-08 08:37:56,385 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-49) Attempting to update VMs/Templates Ovf. >2014-08-08 08:42:52,621 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-30) START, GetFileStatsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 38ec632d >2014-08-08 08:42:52,639 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-30) FINISH, GetFileStatsVDSCommand, return: {test.iso={status=0, ctime=1407430000.0, size=4}}, log id: 38ec632d >2014-08-08 08:42:52,644 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (org.ovirt.thread.pool-8-thread-30) Finished automatic refresh process for ISO file type with success, for storage domain id 63fe3516-4170-402a-9dc8-6d3eab1c7933. >2014-08-08 09:37:52,941 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-29) Principal admin::internal synchronized >2014-08-08 09:37:56,424 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-40) Attempting to update VMs/Templates Ovf. >2014-08-08 09:42:52,655 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-45) START, GetFileStatsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 508fa176 >2014-08-08 09:42:52,671 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-45) FINISH, GetFileStatsVDSCommand, return: {test.iso={status=0, ctime=1407430000.0, size=4}}, log id: 508fa176 >2014-08-08 09:42:52,677 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (org.ovirt.thread.pool-8-thread-45) Finished automatic refresh process for ISO file type with success, for storage domain id 63fe3516-4170-402a-9dc8-6d3eab1c7933. >2014-08-08 10:37:52,947 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-61) Principal admin::internal synchronized >2014-08-08 10:37:56,445 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-55) Attempting to update VMs/Templates Ovf. >2014-08-08 10:42:52,686 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-5) START, GetFileStatsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 4b7e124e >2014-08-08 10:42:52,702 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-5) FINISH, GetFileStatsVDSCommand, return: {test.iso={status=0, ctime=1407430000.0, size=4}}, log id: 4b7e124e >2014-08-08 10:42:52,708 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (org.ovirt.thread.pool-8-thread-5) Finished automatic refresh process for ISO file type with success, for storage domain id 63fe3516-4170-402a-9dc8-6d3eab1c7933. >2014-08-08 11:34:25,181 INFO [org.ovirt.engine.core.bll.aaa.LoginAdminUserCommand] (ajp--127.0.0.1-8702-10) Running command: LoginAdminUserCommand internal: false. >2014-08-08 11:34:25,190 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-10) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-08-08 11:34:38,405 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-1) Running command: LoginUserCommand internal: false. >2014-08-08 11:34:38,416 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-1) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-08-08 11:37:52,953 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-66) [215ad4c4] Principal admin::internal synchronized >2014-08-08 11:37:56,463 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-61) Attempting to update VMs/Templates Ovf. >2014-08-08 11:42:52,721 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-31) START, GetFileStatsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 79c94bd6 >2014-08-08 11:42:52,737 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-31) FINISH, GetFileStatsVDSCommand, return: {test.iso={status=0, ctime=1407430000.0, size=4}}, log id: 79c94bd6 >2014-08-08 11:42:52,748 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (org.ovirt.thread.pool-8-thread-31) Finished automatic refresh process for ISO file type with success, for storage domain id 63fe3516-4170-402a-9dc8-6d3eab1c7933. >2014-08-08 12:37:52,960 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-45) Principal admin::internal synchronized >2014-08-08 12:37:56,495 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-88) Attempting to update VMs/Templates Ovf. >2014-08-08 12:42:52,757 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-10) START, GetFileStatsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 10adc186 >2014-08-08 12:42:52,778 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-10) FINISH, GetFileStatsVDSCommand, return: {test.iso={status=0, ctime=1407430000.0, size=4}}, log id: 10adc186 >2014-08-08 12:42:52,784 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (org.ovirt.thread.pool-8-thread-10) Finished automatic refresh process for ISO file type with success, for storage domain id 63fe3516-4170-402a-9dc8-6d3eab1c7933. >2014-08-08 13:27:07,400 WARN [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-8-thread-8) Executing a command: java.util.concurrent.FutureTask , but note that there are 1 tasks in the queue. >2014-08-08 13:37:52,966 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-60) Principal admin::internal synchronized >2014-08-08 13:37:56,517 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-43) Attempting to update VMs/Templates Ovf. >2014-08-08 13:42:52,793 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-10) START, GetFileStatsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 281ff8c3 >2014-08-08 13:42:52,807 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-10) FINISH, GetFileStatsVDSCommand, return: {test.iso={status=0, ctime=1407430000.0, size=4}}, log id: 281ff8c3 >2014-08-08 13:42:52,813 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (org.ovirt.thread.pool-8-thread-10) Finished automatic refresh process for ISO file type with success, for storage domain id 63fe3516-4170-402a-9dc8-6d3eab1c7933. >2014-08-08 14:37:52,974 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-81) Principal admin::internal synchronized >2014-08-08 14:37:56,537 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-41) Attempting to update VMs/Templates Ovf. >2014-08-08 14:42:52,821 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-40) START, GetFileStatsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 26a95266 >2014-08-08 14:42:52,838 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-40) FINISH, GetFileStatsVDSCommand, return: {test.iso={status=0, ctime=1407430000.0, size=4}}, log id: 26a95266 >2014-08-08 14:42:52,842 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (org.ovirt.thread.pool-8-thread-40) Finished automatic refresh process for ISO file type with success, for storage domain id 63fe3516-4170-402a-9dc8-6d3eab1c7933. >2014-08-08 15:12:46,620 INFO [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (ajp--127.0.0.1-8702-4) [170cbf09] Lock Acquired to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE >, sharedLocks= ] >2014-08-08 15:12:46,684 INFO [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-12) [170cbf09] Running command: DeactivateStorageDomainCommand internal: false. Entities affected : ID: 44559ec0-6bef-491c-bbd6-e67278f37987 Type: Storage >2014-08-08 15:12:46,695 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.DeactivateStorageDomainVDSCommand] (org.ovirt.thread.pool-8-thread-12) [170cbf09] START, DeactivateStorageDomainVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, storageDomainId = 44559ec0-6bef-491c-bbd6-e67278f37987, masterDomainId = 00000000-0000-0000-0000-000000000000, masterVersion = 1), log id: 6a2185c6 >2014-08-08 15:12:46,708 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.DeactivateStorageDomainVDSCommand] (org.ovirt.thread.pool-8-thread-12) [170cbf09] FINISH, DeactivateStorageDomainVDSCommand, log id: 6a2185c6 >2014-08-08 15:12:46,709 INFO [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-12) [170cbf09] Lock freed to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE >, sharedLocks= ] >2014-08-08 15:12:46,723 INFO [org.ovirt.engine.core.bll.storage.AfterDeactivateSingleAsyncOperation] (org.ovirt.thread.pool-8-thread-9) After deactivate treatment vds: rack,pool Default >2014-08-08 15:12:46,725 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-9) START, ConnectStoragePoolVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, masterVersion = 1), log id: a66fe89 >2014-08-08 15:12:47,062 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-9) FINISH, ConnectStoragePoolVDSCommand, log id: a66fe89 >2014-08-08 15:12:47,066 INFO [org.ovirt.engine.core.bll.storage.DisconnectStorageServerConnectionCommand] (org.ovirt.thread.pool-8-thread-12) [3732573e] Running command: DisconnectStorageServerConnectionCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2014-08-08 15:12:47,069 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-12) [3732573e] START, DisconnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: 05047c9a-152e-4ef2-b118-4a1753c1eae0, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35export, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 7add075a >2014-08-08 15:12:47,314 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-12) [3732573e] FINISH, DisconnectStorageServerVDSCommand, return: {05047c9a-152e-4ef2-b118-4a1753c1eae0=0}, log id: 7add075a >2014-08-08 15:12:47,316 INFO [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-12) [3732573e] Domain 44559ec0-6bef-491c-bbd6-e67278f37987 will remain in PreparingForMaintenance status until deactivated on all hosts >2014-08-08 15:12:47,334 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-12) [3732573e] Correlation ID: 170cbf09, Job ID: c7733ca5-9c01-46b5-af1f-e5db0f5a8720, Call Stack: null, Custom Event ID: -1, Message: Storage Domain export (Data Center Default) was deactivated by admin >2014-08-08 15:12:54,819 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-36) Adding domain 44559ec0-6bef-491c-bbd6-e67278f37987 to the domains in maintenance cache >2014-08-08 15:13:02,340 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-11) Moving domain 44559ec0-6bef-491c-bbd6-e67278f37987 to maintenance >2014-08-08 15:37:52,980 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-64) Principal admin::internal synchronized >2014-08-08 15:37:56,558 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-28) Attempting to update VMs/Templates Ovf. >2014-08-08 15:42:52,851 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-50) START, GetFileStatsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 4fe40d4c >2014-08-08 15:42:52,867 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-50) FINISH, GetFileStatsVDSCommand, return: {test.iso={status=0, ctime=1407430000.0, size=4}}, log id: 4fe40d4c >2014-08-08 15:42:52,872 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (org.ovirt.thread.pool-8-thread-50) Finished automatic refresh process for ISO file type with success, for storage domain id 63fe3516-4170-402a-9dc8-6d3eab1c7933. >2014-08-08 16:37:52,986 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-24) Principal admin::internal synchronized >2014-08-08 16:37:56,602 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-78) Attempting to update VMs/Templates Ovf. >2014-08-08 16:42:52,881 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-29) START, GetFileStatsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 7ab331b0 >2014-08-08 16:42:52,897 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-29) FINISH, GetFileStatsVDSCommand, return: {test.iso={status=0, ctime=1407430000.0, size=4}}, log id: 7ab331b0 >2014-08-08 16:42:52,902 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (org.ovirt.thread.pool-8-thread-29) Finished automatic refresh process for ISO file type with success, for storage domain id 63fe3516-4170-402a-9dc8-6d3eab1c7933. >2014-08-08 17:37:52,995 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-29) Principal admin::internal synchronized >2014-08-08 17:37:56,628 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-3) Attempting to update VMs/Templates Ovf. >2014-08-08 17:42:52,913 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-43) START, GetFileStatsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 768ea522 >2014-08-08 17:42:52,931 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-43) FINISH, GetFileStatsVDSCommand, return: {test.iso={status=0, ctime=1407430000.0, size=4}}, log id: 768ea522 >2014-08-08 17:42:52,937 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (org.ovirt.thread.pool-8-thread-43) Finished automatic refresh process for ISO file type with success, for storage domain id 63fe3516-4170-402a-9dc8-6d3eab1c7933. >2014-08-08 18:37:53,004 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-57) Principal admin::internal synchronized >2014-08-08 18:37:56,661 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-7) Attempting to update VMs/Templates Ovf. >2014-08-08 18:39:56,867 WARN [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-8-thread-38) Executing a command: java.util.concurrent.FutureTask , but note that there are 1 tasks in the queue. >2014-08-08 18:42:52,948 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-13) START, GetFileStatsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 608b195c >2014-08-08 18:42:52,963 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-13) FINISH, GetFileStatsVDSCommand, return: {test.iso={status=0, ctime=1407430000.0, size=4}}, log id: 608b195c >2014-08-08 18:42:52,969 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (org.ovirt.thread.pool-8-thread-13) Finished automatic refresh process for ISO file type with success, for storage domain id 63fe3516-4170-402a-9dc8-6d3eab1c7933. >2014-08-08 19:37:53,014 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-99) Principal admin::internal synchronized >2014-08-08 19:37:56,685 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-61) Attempting to update VMs/Templates Ovf. >2014-08-08 19:42:52,978 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-40) START, GetFileStatsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 555199ad >2014-08-08 19:42:52,995 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-40) FINISH, GetFileStatsVDSCommand, return: {test.iso={status=0, ctime=1407430000.0, size=4}}, log id: 555199ad >2014-08-08 19:42:53,001 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (org.ovirt.thread.pool-8-thread-40) Finished automatic refresh process for ISO file type with success, for storage domain id 63fe3516-4170-402a-9dc8-6d3eab1c7933. >2014-08-08 20:37:53,021 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-53) Principal admin::internal synchronized >2014-08-08 20:37:56,720 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-63) Attempting to update VMs/Templates Ovf. >2014-08-08 20:42:53,012 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-5) START, GetFileStatsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 12b46b69 >2014-08-08 20:42:53,028 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-5) FINISH, GetFileStatsVDSCommand, return: {test.iso={status=0, ctime=1407430000.0, size=4}}, log id: 12b46b69 >2014-08-08 20:42:53,034 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (org.ovirt.thread.pool-8-thread-5) Finished automatic refresh process for ISO file type with success, for storage domain id 63fe3516-4170-402a-9dc8-6d3eab1c7933. >2014-08-08 21:37:53,026 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-47) Principal admin::internal synchronized >2014-08-08 21:37:56,740 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-2) Attempting to update VMs/Templates Ovf. >2014-08-08 21:42:53,044 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-40) START, GetFileStatsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 6b006980 >2014-08-08 21:42:53,062 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-40) FINISH, GetFileStatsVDSCommand, return: {test.iso={status=0, ctime=1407430000.0, size=4}}, log id: 6b006980 >2014-08-08 21:42:53,068 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (org.ovirt.thread.pool-8-thread-40) Finished automatic refresh process for ISO file type with success, for storage domain id 63fe3516-4170-402a-9dc8-6d3eab1c7933. >2014-08-08 22:37:53,032 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-78) Principal admin::internal synchronized >2014-08-08 22:37:56,779 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-30) Attempting to update VMs/Templates Ovf. >2014-08-08 22:42:53,077 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-26) START, GetFileStatsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 1d66e654 >2014-08-08 22:42:53,093 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-26) FINISH, GetFileStatsVDSCommand, return: {test.iso={status=0, ctime=1407430000.0, size=4}}, log id: 1d66e654 >2014-08-08 22:42:53,098 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (org.ovirt.thread.pool-8-thread-26) Finished automatic refresh process for ISO file type with success, for storage domain id 63fe3516-4170-402a-9dc8-6d3eab1c7933. >2014-08-08 23:37:53,035 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-31) Principal admin::internal synchronized >2014-08-08 23:37:56,803 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-37) Attempting to update VMs/Templates Ovf. >2014-08-08 23:42:53,108 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-44) START, GetFileStatsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 717c9cb4 >2014-08-08 23:42:53,125 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-44) FINISH, GetFileStatsVDSCommand, return: {test.iso={status=0, ctime=1407430000.0, size=4}}, log id: 717c9cb4 >2014-08-08 23:42:53,130 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (org.ovirt.thread.pool-8-thread-44) Finished automatic refresh process for ISO file type with success, for storage domain id 63fe3516-4170-402a-9dc8-6d3eab1c7933. >2014-08-09 00:37:53,043 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-4) Principal admin::internal synchronized >2014-08-09 00:37:56,823 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-52) Attempting to update VMs/Templates Ovf. >2014-08-09 00:42:53,140 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-19) START, GetFileStatsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 47cc8e8c >2014-08-09 00:42:53,158 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-19) FINISH, GetFileStatsVDSCommand, return: {test.iso={status=0, ctime=1407430000.0, size=4}}, log id: 47cc8e8c >2014-08-09 00:42:53,165 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (org.ovirt.thread.pool-8-thread-19) Finished automatic refresh process for ISO file type with success, for storage domain id 63fe3516-4170-402a-9dc8-6d3eab1c7933. >2014-08-09 01:37:53,051 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-22) Principal admin::internal synchronized >2014-08-09 01:37:56,857 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-96) Attempting to update VMs/Templates Ovf. >2014-08-09 01:42:53,174 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-19) START, GetFileStatsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: fab49ec >2014-08-09 01:42:53,192 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-19) FINISH, GetFileStatsVDSCommand, return: {test.iso={status=0, ctime=1407430000.0, size=4}}, log id: fab49ec >2014-08-09 01:42:53,199 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (org.ovirt.thread.pool-8-thread-19) Finished automatic refresh process for ISO file type with success, for storage domain id 63fe3516-4170-402a-9dc8-6d3eab1c7933. >2014-08-09 02:37:53,059 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-11) Principal admin::internal synchronized >2014-08-09 02:37:56,878 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-5) Attempting to update VMs/Templates Ovf. >2014-08-09 02:42:53,212 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-48) START, GetFileStatsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 4f761905 >2014-08-09 02:42:53,228 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-48) FINISH, GetFileStatsVDSCommand, return: {test.iso={status=0, ctime=1407430000.0, size=4}}, log id: 4f761905 >2014-08-09 02:42:53,235 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (org.ovirt.thread.pool-8-thread-48) Finished automatic refresh process for ISO file type with success, for storage domain id 63fe3516-4170-402a-9dc8-6d3eab1c7933. >2014-08-09 03:35:35,003 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-17) Start deleteAgedOutAuditLogs >2014-08-09 03:35:35,003 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-70) Start deleteAgedOutCommandEntities >2014-08-09 03:35:35,010 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-70) Finished deleteAgedOutCommandEntities >2014-08-09 03:35:35,011 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-17) Finished deleteAgedOutAuditLogs >2014-08-09 03:37:53,064 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-6) [46247d52] Principal admin::internal synchronized >2014-08-09 03:37:56,896 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-58) Attempting to update VMs/Templates Ovf. >2014-08-09 03:42:53,244 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-26) START, GetFileStatsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 6c97c0ab >2014-08-09 03:42:53,260 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-26) FINISH, GetFileStatsVDSCommand, return: {test.iso={status=0, ctime=1407430000.0, size=4}}, log id: 6c97c0ab >2014-08-09 03:42:53,266 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (org.ovirt.thread.pool-8-thread-26) Finished automatic refresh process for ISO file type with success, for storage domain id 63fe3516-4170-402a-9dc8-6d3eab1c7933. >2014-08-09 04:37:53,073 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-37) Principal admin::internal synchronized >2014-08-09 04:37:56,918 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-81) Attempting to update VMs/Templates Ovf. >2014-08-09 04:42:53,277 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-16) START, GetFileStatsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 7cd1d4f4 >2014-08-09 04:42:53,299 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-16) FINISH, GetFileStatsVDSCommand, return: {test.iso={status=0, ctime=1407430000.0, size=4}}, log id: 7cd1d4f4 >2014-08-09 04:42:53,308 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (org.ovirt.thread.pool-8-thread-16) Finished automatic refresh process for ISO file type with success, for storage domain id 63fe3516-4170-402a-9dc8-6d3eab1c7933. >2014-08-09 05:37:53,079 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-49) Principal admin::internal synchronized >2014-08-09 05:37:56,950 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-26) Attempting to update VMs/Templates Ovf. >2014-08-09 05:42:53,319 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-18) START, GetFileStatsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 30eb1cc7 >2014-08-09 05:42:53,336 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-18) FINISH, GetFileStatsVDSCommand, return: {test.iso={status=0, ctime=1407430000.0, size=4}}, log id: 30eb1cc7 >2014-08-09 05:42:53,341 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (org.ovirt.thread.pool-8-thread-18) Finished automatic refresh process for ISO file type with success, for storage domain id 63fe3516-4170-402a-9dc8-6d3eab1c7933. >2014-08-09 06:37:53,089 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-33) Principal admin::internal synchronized >2014-08-09 06:37:56,970 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-72) Attempting to update VMs/Templates Ovf. >2014-08-09 06:42:53,354 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-42) START, GetFileStatsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 4df0ce86 >2014-08-09 06:42:53,372 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-42) FINISH, GetFileStatsVDSCommand, return: {test.iso={status=0, ctime=1407430000.0, size=4}}, log id: 4df0ce86 >2014-08-09 06:42:53,379 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (org.ovirt.thread.pool-8-thread-42) Finished automatic refresh process for ISO file type with success, for storage domain id 63fe3516-4170-402a-9dc8-6d3eab1c7933. >2014-08-09 07:37:53,094 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-68) Principal admin::internal synchronized >2014-08-09 07:37:57,002 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-33) Attempting to update VMs/Templates Ovf. >2014-08-09 07:42:53,390 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-37) START, GetFileStatsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 70fe66e2 >2014-08-09 07:42:53,408 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-37) FINISH, GetFileStatsVDSCommand, return: {test.iso={status=0, ctime=1407430000.0, size=4}}, log id: 70fe66e2 >2014-08-09 07:42:53,414 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (org.ovirt.thread.pool-8-thread-37) Finished automatic refresh process for ISO file type with success, for storage domain id 63fe3516-4170-402a-9dc8-6d3eab1c7933. >2014-08-09 08:37:53,105 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-86) Principal admin::internal synchronized >2014-08-09 08:37:57,023 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-70) Attempting to update VMs/Templates Ovf. >2014-08-09 08:42:53,421 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-28) START, GetFileStatsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: bb1a5d9 >2014-08-09 08:42:53,439 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-28) FINISH, GetFileStatsVDSCommand, return: {test.iso={status=0, ctime=1407430000.0, size=4}}, log id: bb1a5d9 >2014-08-09 08:42:53,444 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (org.ovirt.thread.pool-8-thread-28) Finished automatic refresh process for ISO file type with success, for storage domain id 63fe3516-4170-402a-9dc8-6d3eab1c7933. >2014-08-09 08:46:35,420 WARN [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-8-thread-46) Executing a command: java.util.concurrent.FutureTask , but note that there are 1 tasks in the queue. >2014-08-09 09:37:53,113 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-61) Principal admin::internal synchronized >2014-08-09 09:37:57,041 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-71) Attempting to update VMs/Templates Ovf. >2014-08-09 09:42:53,454 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-33) START, GetFileStatsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 6943cb34 >2014-08-09 09:42:53,471 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-33) FINISH, GetFileStatsVDSCommand, return: {test.iso={status=0, ctime=1407430000.0, size=4}}, log id: 6943cb34 >2014-08-09 09:42:53,476 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (org.ovirt.thread.pool-8-thread-33) Finished automatic refresh process for ISO file type with success, for storage domain id 63fe3516-4170-402a-9dc8-6d3eab1c7933. >2014-08-09 10:37:53,121 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-3) Principal admin::internal synchronized >2014-08-09 10:37:57,078 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-93) Attempting to update VMs/Templates Ovf. >2014-08-09 10:42:53,485 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-27) START, GetFileStatsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 1c0be4c5 >2014-08-09 10:42:53,503 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-27) FINISH, GetFileStatsVDSCommand, return: {test.iso={status=0, ctime=1407430000.0, size=4}}, log id: 1c0be4c5 >2014-08-09 10:42:53,510 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (org.ovirt.thread.pool-8-thread-27) Finished automatic refresh process for ISO file type with success, for storage domain id 63fe3516-4170-402a-9dc8-6d3eab1c7933. >2014-08-09 11:37:53,131 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-6) [46247d52] Principal admin::internal synchronized >2014-08-09 11:37:57,099 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-41) Attempting to update VMs/Templates Ovf. >2014-08-09 11:42:53,521 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-1) START, GetFileStatsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 5746ee64 >2014-08-09 11:42:53,539 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-1) FINISH, GetFileStatsVDSCommand, return: {test.iso={status=0, ctime=1407430000.0, size=4}}, log id: 5746ee64 >2014-08-09 11:42:53,546 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (org.ovirt.thread.pool-8-thread-1) Finished automatic refresh process for ISO file type with success, for storage domain id 63fe3516-4170-402a-9dc8-6d3eab1c7933. >2014-08-09 12:37:53,139 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-10) Principal admin::internal synchronized >2014-08-09 12:37:57,131 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-32) Attempting to update VMs/Templates Ovf. >2014-08-09 12:42:53,563 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-40) START, GetFileStatsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 44a7aa9b >2014-08-09 12:42:53,579 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-40) FINISH, GetFileStatsVDSCommand, return: {test.iso={status=0, ctime=1407430000.0, size=4}}, log id: 44a7aa9b >2014-08-09 12:42:53,585 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (org.ovirt.thread.pool-8-thread-40) Finished automatic refresh process for ISO file type with success, for storage domain id 63fe3516-4170-402a-9dc8-6d3eab1c7933. >2014-08-09 13:37:53,146 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-76) Principal admin::internal synchronized >2014-08-09 13:37:57,153 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-10) Attempting to update VMs/Templates Ovf. >2014-08-09 13:42:53,597 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-28) START, GetFileStatsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 5dcdd181 >2014-08-09 13:42:53,614 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-28) FINISH, GetFileStatsVDSCommand, return: {test.iso={status=0, ctime=1407430000.0, size=4}}, log id: 5dcdd181 >2014-08-09 13:42:53,620 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (org.ovirt.thread.pool-8-thread-28) Finished automatic refresh process for ISO file type with success, for storage domain id 63fe3516-4170-402a-9dc8-6d3eab1c7933. >2014-08-09 14:37:53,154 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-47) Principal admin::internal synchronized >2014-08-09 14:37:57,175 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-30) Attempting to update VMs/Templates Ovf. >2014-08-09 14:42:53,628 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-7) START, GetFileStatsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 6e87be8b >2014-08-09 14:42:53,645 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-7) FINISH, GetFileStatsVDSCommand, return: {test.iso={status=0, ctime=1407430000.0, size=4}}, log id: 6e87be8b >2014-08-09 14:42:53,650 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (org.ovirt.thread.pool-8-thread-7) Finished automatic refresh process for ISO file type with success, for storage domain id 63fe3516-4170-402a-9dc8-6d3eab1c7933. >2014-08-09 15:37:53,161 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-90) Principal admin::internal synchronized >2014-08-09 15:37:57,210 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-52) Attempting to update VMs/Templates Ovf. >2014-08-09 15:42:53,659 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-50) START, GetFileStatsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 28bd6d77 >2014-08-09 15:42:53,674 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-50) FINISH, GetFileStatsVDSCommand, return: {test.iso={status=0, ctime=1407430000.0, size=4}}, log id: 28bd6d77 >2014-08-09 15:42:53,679 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (org.ovirt.thread.pool-8-thread-50) Finished automatic refresh process for ISO file type with success, for storage domain id 63fe3516-4170-402a-9dc8-6d3eab1c7933. >2014-08-09 16:37:53,166 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-11) Principal admin::internal synchronized >2014-08-09 16:37:57,233 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-91) Attempting to update VMs/Templates Ovf. >2014-08-09 16:42:53,691 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-11) START, GetFileStatsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 7d65f981 >2014-08-09 16:42:53,707 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-11) FINISH, GetFileStatsVDSCommand, return: {test.iso={status=0, ctime=1407430000.0, size=4}}, log id: 7d65f981 >2014-08-09 16:42:53,713 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (org.ovirt.thread.pool-8-thread-11) Finished automatic refresh process for ISO file type with success, for storage domain id 63fe3516-4170-402a-9dc8-6d3eab1c7933. >2014-08-09 17:37:53,172 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-37) Principal admin::internal synchronized >2014-08-09 17:37:57,269 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-61) Attempting to update VMs/Templates Ovf. >2014-08-09 17:42:53,722 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-3) START, GetFileStatsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 5298f950 >2014-08-09 17:42:53,738 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-3) FINISH, GetFileStatsVDSCommand, return: {test.iso={status=0, ctime=1407430000.0, size=4}}, log id: 5298f950 >2014-08-09 17:42:53,743 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (org.ovirt.thread.pool-8-thread-3) Finished automatic refresh process for ISO file type with success, for storage domain id 63fe3516-4170-402a-9dc8-6d3eab1c7933. >2014-08-09 18:37:53,177 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-81) Principal admin::internal synchronized >2014-08-09 18:37:57,289 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-5) Attempting to update VMs/Templates Ovf. >2014-08-09 18:42:53,754 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-48) START, GetFileStatsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 712a41c7 >2014-08-09 18:42:53,772 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-48) FINISH, GetFileStatsVDSCommand, return: {test.iso={status=0, ctime=1407430000.0, size=4}}, log id: 712a41c7 >2014-08-09 18:42:53,780 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (org.ovirt.thread.pool-8-thread-48) Finished automatic refresh process for ISO file type with success, for storage domain id 63fe3516-4170-402a-9dc8-6d3eab1c7933. >2014-08-09 19:37:53,182 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-60) Principal admin::internal synchronized >2014-08-09 19:37:57,309 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-56) Attempting to update VMs/Templates Ovf. >2014-08-09 19:42:53,788 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-46) [215ad4c4] START, GetFileStatsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 68e2d83c >2014-08-09 19:42:53,804 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-46) [215ad4c4] FINISH, GetFileStatsVDSCommand, return: {test.iso={status=0, ctime=1407430000.0, size=4}}, log id: 68e2d83c >2014-08-09 19:42:53,811 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (org.ovirt.thread.pool-8-thread-46) [215ad4c4] Finished automatic refresh process for ISO file type with success, for storage domain id 63fe3516-4170-402a-9dc8-6d3eab1c7933. >2014-08-09 20:37:53,187 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-64) Principal admin::internal synchronized >2014-08-09 20:37:57,328 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-33) Attempting to update VMs/Templates Ovf. >2014-08-09 20:42:53,820 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-18) START, GetFileStatsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 5f516bee >2014-08-09 20:42:53,836 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-18) FINISH, GetFileStatsVDSCommand, return: {test.iso={status=0, ctime=1407430000.0, size=4}}, log id: 5f516bee >2014-08-09 20:42:53,841 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (org.ovirt.thread.pool-8-thread-18) Finished automatic refresh process for ISO file type with success, for storage domain id 63fe3516-4170-402a-9dc8-6d3eab1c7933. >2014-08-09 21:37:53,192 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-76) Principal admin::internal synchronized >2014-08-09 21:37:57,363 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-86) Attempting to update VMs/Templates Ovf. >2014-08-09 21:42:53,854 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-25) START, GetFileStatsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 2020d46f >2014-08-09 21:42:53,869 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-25) FINISH, GetFileStatsVDSCommand, return: {test.iso={status=0, ctime=1407430000.0, size=4}}, log id: 2020d46f >2014-08-09 21:42:53,875 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (org.ovirt.thread.pool-8-thread-25) Finished automatic refresh process for ISO file type with success, for storage domain id 63fe3516-4170-402a-9dc8-6d3eab1c7933. >2014-08-09 22:37:53,198 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-55) Principal admin::internal synchronized >2014-08-09 22:37:57,384 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-65) Attempting to update VMs/Templates Ovf. >2014-08-09 22:42:53,884 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-14) START, GetFileStatsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 51be8099 >2014-08-09 22:42:53,899 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-14) FINISH, GetFileStatsVDSCommand, return: {test.iso={status=0, ctime=1407430000.0, size=4}}, log id: 51be8099 >2014-08-09 22:42:53,906 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (org.ovirt.thread.pool-8-thread-14) Finished automatic refresh process for ISO file type with success, for storage domain id 63fe3516-4170-402a-9dc8-6d3eab1c7933. >2014-08-09 23:37:53,206 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-53) Principal admin::internal synchronized >2014-08-09 23:37:57,405 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-95) Attempting to update VMs/Templates Ovf. >2014-08-09 23:42:53,915 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-35) START, GetFileStatsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 4ae3e649 >2014-08-09 23:42:53,933 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-35) FINISH, GetFileStatsVDSCommand, return: {test.iso={status=0, ctime=1407430000.0, size=4}}, log id: 4ae3e649 >2014-08-09 23:42:53,941 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (org.ovirt.thread.pool-8-thread-35) Finished automatic refresh process for ISO file type with success, for storage domain id 63fe3516-4170-402a-9dc8-6d3eab1c7933. >2014-08-10 00:37:53,214 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-57) Principal admin::internal synchronized >2014-08-10 00:37:57,423 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-100) Attempting to update VMs/Templates Ovf. >2014-08-10 00:42:53,952 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-49) [7f085c60] START, GetFileStatsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 504970c3 >2014-08-10 00:42:53,967 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-49) [7f085c60] FINISH, GetFileStatsVDSCommand, return: {test.iso={status=0, ctime=1407430000.0, size=4}}, log id: 504970c3 >2014-08-10 00:42:53,973 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (org.ovirt.thread.pool-8-thread-49) [7f085c60] Finished automatic refresh process for ISO file type with success, for storage domain id 63fe3516-4170-402a-9dc8-6d3eab1c7933. >2014-08-10 01:37:53,221 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-48) Principal admin::internal synchronized >2014-08-10 01:37:57,462 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-57) Attempting to update VMs/Templates Ovf. >2014-08-10 01:42:53,986 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-25) START, GetFileStatsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 7956fbf1 >2014-08-10 01:42:54,003 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-25) FINISH, GetFileStatsVDSCommand, return: {test.iso={status=0, ctime=1407430000.0, size=4}}, log id: 7956fbf1 >2014-08-10 01:42:54,009 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (org.ovirt.thread.pool-8-thread-25) Finished automatic refresh process for ISO file type with success, for storage domain id 63fe3516-4170-402a-9dc8-6d3eab1c7933. >2014-08-10 02:37:53,227 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-87) Principal admin::internal synchronized >2014-08-10 02:37:57,484 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-30) Attempting to update VMs/Templates Ovf. >2014-08-10 02:42:54,017 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-2) START, GetFileStatsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 67158e84 >2014-08-10 02:42:54,033 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-2) FINISH, GetFileStatsVDSCommand, return: {test.iso={status=0, ctime=1407430000.0, size=4}}, log id: 67158e84 >2014-08-10 02:42:54,037 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (org.ovirt.thread.pool-8-thread-2) Finished automatic refresh process for ISO file type with success, for storage domain id 63fe3516-4170-402a-9dc8-6d3eab1c7933. >2014-08-10 03:35:35,001 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-3) Start deleteAgedOutAuditLogs >2014-08-10 03:35:35,003 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-42) Start deleteAgedOutCommandEntities >2014-08-10 03:35:35,011 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-3) Finished deleteAgedOutAuditLogs >2014-08-10 03:35:35,015 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-42) Finished deleteAgedOutCommandEntities >2014-08-10 03:37:53,236 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-64) Principal admin::internal synchronized >2014-08-10 03:37:57,512 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-27) Attempting to update VMs/Templates Ovf. >2014-08-10 03:42:54,044 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-3) START, GetFileStatsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 62baab2b >2014-08-10 03:42:54,061 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-3) FINISH, GetFileStatsVDSCommand, return: {test.iso={status=0, ctime=1407430000.0, size=4}}, log id: 62baab2b >2014-08-10 03:42:54,066 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (org.ovirt.thread.pool-8-thread-3) Finished automatic refresh process for ISO file type with success, for storage domain id 63fe3516-4170-402a-9dc8-6d3eab1c7933. >2014-08-10 04:37:53,244 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-62) Principal admin::internal synchronized >2014-08-10 04:37:57,531 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-46) Attempting to update VMs/Templates Ovf. >2014-08-10 04:42:54,074 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-20) START, GetFileStatsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 27e3011b >2014-08-10 04:42:54,091 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-20) FINISH, GetFileStatsVDSCommand, return: {test.iso={status=0, ctime=1407430000.0, size=4}}, log id: 27e3011b >2014-08-10 04:42:54,096 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (org.ovirt.thread.pool-8-thread-20) Finished automatic refresh process for ISO file type with success, for storage domain id 63fe3516-4170-402a-9dc8-6d3eab1c7933. >2014-08-10 05:37:53,252 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-23) Principal admin::internal synchronized >2014-08-10 05:37:57,554 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-75) Attempting to update VMs/Templates Ovf. >2014-08-10 05:42:54,104 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-40) START, GetFileStatsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 560db072 >2014-08-10 05:42:54,121 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-40) FINISH, GetFileStatsVDSCommand, return: {test.iso={status=0, ctime=1407430000.0, size=4}}, log id: 560db072 >2014-08-10 05:42:54,126 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (org.ovirt.thread.pool-8-thread-40) Finished automatic refresh process for ISO file type with success, for storage domain id 63fe3516-4170-402a-9dc8-6d3eab1c7933. >2014-08-10 06:37:53,261 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-61) Principal admin::internal synchronized >2014-08-10 06:37:57,591 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-50) Attempting to update VMs/Templates Ovf. >2014-08-10 06:42:54,133 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-47) START, GetFileStatsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 53cd5519 >2014-08-10 06:42:54,149 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-47) FINISH, GetFileStatsVDSCommand, return: {test.iso={status=0, ctime=1407430000.0, size=4}}, log id: 53cd5519 >2014-08-10 06:42:54,153 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (org.ovirt.thread.pool-8-thread-47) Finished automatic refresh process for ISO file type with success, for storage domain id 63fe3516-4170-402a-9dc8-6d3eab1c7933. >2014-08-10 07:37:53,270 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-1) Principal admin::internal synchronized >2014-08-10 07:37:57,618 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-52) Attempting to update VMs/Templates Ovf. >2014-08-10 07:42:54,165 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-7) START, GetFileStatsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 5f28cd5d >2014-08-10 07:42:54,181 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-7) FINISH, GetFileStatsVDSCommand, return: {test.iso={status=0, ctime=1407430000.0, size=4}}, log id: 5f28cd5d >2014-08-10 07:42:54,188 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (org.ovirt.thread.pool-8-thread-7) Finished automatic refresh process for ISO file type with success, for storage domain id 63fe3516-4170-402a-9dc8-6d3eab1c7933. >2014-08-10 08:37:53,279 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-42) Principal admin::internal synchronized >2014-08-10 08:37:57,640 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-1) Attempting to update VMs/Templates Ovf. >2014-08-10 08:42:54,200 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-2) START, GetFileStatsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 7781ebf0 >2014-08-10 08:42:54,216 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-2) FINISH, GetFileStatsVDSCommand, return: {test.iso={status=0, ctime=1407430000.0, size=4}}, log id: 7781ebf0 >2014-08-10 08:42:54,222 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (org.ovirt.thread.pool-8-thread-2) Finished automatic refresh process for ISO file type with success, for storage domain id 63fe3516-4170-402a-9dc8-6d3eab1c7933. >2014-08-10 09:37:53,288 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-92) Principal admin::internal synchronized >2014-08-10 09:37:57,673 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-30) Attempting to update VMs/Templates Ovf. >2014-08-10 09:42:54,233 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-34) START, GetFileStatsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 1006ed2a >2014-08-10 09:42:54,320 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-34) FINISH, GetFileStatsVDSCommand, return: {test.iso={status=0, ctime=1407430000.0, size=4}}, log id: 1006ed2a >2014-08-10 09:42:54,325 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (org.ovirt.thread.pool-8-thread-34) Finished automatic refresh process for ISO file type with success, for storage domain id 63fe3516-4170-402a-9dc8-6d3eab1c7933. >2014-08-10 10:37:53,294 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-7) Principal admin::internal synchronized >2014-08-10 10:37:57,710 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-69) Attempting to update VMs/Templates Ovf. >2014-08-10 10:42:54,338 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-31) START, GetFileStatsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 7206f594 >2014-08-10 10:42:54,355 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-31) FINISH, GetFileStatsVDSCommand, return: {test.iso={status=0, ctime=1407430000.0, size=4}}, log id: 7206f594 >2014-08-10 10:42:54,362 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (org.ovirt.thread.pool-8-thread-31) Finished automatic refresh process for ISO file type with success, for storage domain id 63fe3516-4170-402a-9dc8-6d3eab1c7933. >2014-08-10 11:37:53,300 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-34) Principal admin::internal synchronized >2014-08-10 11:37:57,747 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-7) Attempting to update VMs/Templates Ovf. >2014-08-10 11:42:54,372 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-35) START, GetFileStatsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 5f3e0d38 >2014-08-10 11:42:54,389 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-35) FINISH, GetFileStatsVDSCommand, return: {test.iso={status=0, ctime=1407430000.0, size=4}}, log id: 5f3e0d38 >2014-08-10 11:42:54,393 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (org.ovirt.thread.pool-8-thread-35) Finished automatic refresh process for ISO file type with success, for storage domain id 63fe3516-4170-402a-9dc8-6d3eab1c7933. >2014-08-10 12:37:53,310 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-51) Principal admin::internal synchronized >2014-08-10 12:37:57,771 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-79) Attempting to update VMs/Templates Ovf. >2014-08-10 12:42:54,404 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-31) START, GetFileStatsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 492a8558 >2014-08-10 12:42:55,310 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-31) FINISH, GetFileStatsVDSCommand, return: {test.iso={status=0, ctime=1407430000.0, size=4}}, log id: 492a8558 >2014-08-10 12:42:55,315 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (org.ovirt.thread.pool-8-thread-31) Finished automatic refresh process for ISO file type with success, for storage domain id 63fe3516-4170-402a-9dc8-6d3eab1c7933. >2014-08-10 13:37:53,316 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-52) Principal admin::internal synchronized >2014-08-10 13:37:57,791 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-82) Attempting to update VMs/Templates Ovf. >2014-08-10 13:42:55,327 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-41) START, GetFileStatsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 1f1f2d28 >2014-08-10 13:42:55,405 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-41) FINISH, GetFileStatsVDSCommand, return: {test.iso={status=0, ctime=1407430000.0, size=4}}, log id: 1f1f2d28 >2014-08-10 13:42:55,411 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (org.ovirt.thread.pool-8-thread-41) Finished automatic refresh process for ISO file type with success, for storage domain id 63fe3516-4170-402a-9dc8-6d3eab1c7933. >2014-08-10 14:37:53,326 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-20) Principal admin::internal synchronized >2014-08-10 14:37:57,810 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-72) Attempting to update VMs/Templates Ovf. >2014-08-10 14:42:55,422 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-38) START, GetFileStatsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: f601290 >2014-08-10 14:42:55,441 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-38) FINISH, GetFileStatsVDSCommand, return: {test.iso={status=0, ctime=1407430000.0, size=4}}, log id: f601290 >2014-08-10 14:42:55,447 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (org.ovirt.thread.pool-8-thread-38) Finished automatic refresh process for ISO file type with success, for storage domain id 63fe3516-4170-402a-9dc8-6d3eab1c7933. >2014-08-10 15:37:53,337 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-67) Principal admin::internal synchronized >2014-08-10 15:37:57,849 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-80) Attempting to update VMs/Templates Ovf. >2014-08-10 15:42:55,461 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-19) START, GetFileStatsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 580ce1 >2014-08-10 15:42:55,478 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-19) FINISH, GetFileStatsVDSCommand, return: {test.iso={status=0, ctime=1407430000.0, size=4}}, log id: 580ce1 >2014-08-10 15:42:55,484 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (org.ovirt.thread.pool-8-thread-19) Finished automatic refresh process for ISO file type with success, for storage domain id 63fe3516-4170-402a-9dc8-6d3eab1c7933. >2014-08-10 16:37:53,343 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-48) Principal admin::internal synchronized >2014-08-10 16:37:57,889 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-28) Attempting to update VMs/Templates Ovf. >2014-08-10 16:42:55,490 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-50) START, GetFileStatsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 6af9a289 >2014-08-10 16:42:55,507 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-50) FINISH, GetFileStatsVDSCommand, return: {test.iso={status=0, ctime=1407430000.0, size=4}}, log id: 6af9a289 >2014-08-10 16:42:55,512 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (org.ovirt.thread.pool-8-thread-50) Finished automatic refresh process for ISO file type with success, for storage domain id 63fe3516-4170-402a-9dc8-6d3eab1c7933. >2014-08-10 17:37:53,349 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-2) Principal admin::internal synchronized >2014-08-10 17:37:57,908 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-59) Attempting to update VMs/Templates Ovf. >2014-08-10 17:42:55,519 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-12) START, GetFileStatsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 377fc645 >2014-08-10 17:42:55,536 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-12) FINISH, GetFileStatsVDSCommand, return: {test.iso={status=0, ctime=1407430000.0, size=4}}, log id: 377fc645 >2014-08-10 17:42:55,541 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (org.ovirt.thread.pool-8-thread-12) Finished automatic refresh process for ISO file type with success, for storage domain id 63fe3516-4170-402a-9dc8-6d3eab1c7933. >2014-08-10 18:37:53,358 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-53) Principal admin::internal synchronized >2014-08-10 18:37:57,927 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-90) Attempting to update VMs/Templates Ovf. >2014-08-10 18:42:55,554 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-29) START, GetFileStatsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 4a563585 >2014-08-10 18:42:55,572 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-29) FINISH, GetFileStatsVDSCommand, return: {test.iso={status=0, ctime=1407430000.0, size=4}}, log id: 4a563585 >2014-08-10 18:42:55,580 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (org.ovirt.thread.pool-8-thread-29) Finished automatic refresh process for ISO file type with success, for storage domain id 63fe3516-4170-402a-9dc8-6d3eab1c7933. >2014-08-10 19:37:53,366 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-74) Principal admin::internal synchronized >2014-08-10 19:37:57,960 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-83) Attempting to update VMs/Templates Ovf. >2014-08-10 19:42:55,593 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-2) START, GetFileStatsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 5371f179 >2014-08-10 19:42:55,610 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-2) FINISH, GetFileStatsVDSCommand, return: {test.iso={status=0, ctime=1407430000.0, size=4}}, log id: 5371f179 >2014-08-10 19:42:55,616 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (org.ovirt.thread.pool-8-thread-2) Finished automatic refresh process for ISO file type with success, for storage domain id 63fe3516-4170-402a-9dc8-6d3eab1c7933. >2014-08-10 20:37:53,375 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-65) Principal admin::internal synchronized >2014-08-10 20:37:57,999 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-15) [6fcc1be5] Attempting to update VMs/Templates Ovf. >2014-08-10 20:42:55,628 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-4) START, GetFileStatsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 16eef8f3 >2014-08-10 20:42:55,644 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-4) FINISH, GetFileStatsVDSCommand, return: {test.iso={status=0, ctime=1407430000.0, size=4}}, log id: 16eef8f3 >2014-08-10 20:42:55,651 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (org.ovirt.thread.pool-8-thread-4) Finished automatic refresh process for ISO file type with success, for storage domain id 63fe3516-4170-402a-9dc8-6d3eab1c7933. >2014-08-10 21:37:53,384 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-69) Principal admin::internal synchronized >2014-08-10 21:37:58,034 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-93) Attempting to update VMs/Templates Ovf. >2014-08-10 21:42:55,664 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-35) START, GetFileStatsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 65fe5038 >2014-08-10 21:42:55,680 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-35) FINISH, GetFileStatsVDSCommand, return: {test.iso={status=0, ctime=1407430000.0, size=4}}, log id: 65fe5038 >2014-08-10 21:42:55,686 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (org.ovirt.thread.pool-8-thread-35) Finished automatic refresh process for ISO file type with success, for storage domain id 63fe3516-4170-402a-9dc8-6d3eab1c7933. >2014-08-10 22:37:53,391 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-61) Principal admin::internal synchronized >2014-08-10 22:37:58,073 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-100) Attempting to update VMs/Templates Ovf. >2014-08-10 22:42:55,695 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-50) START, GetFileStatsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 32dbfb15 >2014-08-10 22:42:55,758 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-50) FINISH, GetFileStatsVDSCommand, return: {test.iso={status=0, ctime=1407430000.0, size=4}}, log id: 32dbfb15 >2014-08-10 22:42:55,766 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (org.ovirt.thread.pool-8-thread-50) Finished automatic refresh process for ISO file type with success, for storage domain id 63fe3516-4170-402a-9dc8-6d3eab1c7933. >2014-08-10 23:37:53,398 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-27) Principal admin::internal synchronized >2014-08-10 23:37:58,091 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-30) Attempting to update VMs/Templates Ovf. >2014-08-10 23:42:55,779 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-20) START, GetFileStatsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 707a5776 >2014-08-10 23:42:55,796 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-20) FINISH, GetFileStatsVDSCommand, return: {test.iso={status=0, ctime=1407430000.0, size=4}}, log id: 707a5776 >2014-08-10 23:42:55,803 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (org.ovirt.thread.pool-8-thread-20) Finished automatic refresh process for ISO file type with success, for storage domain id 63fe3516-4170-402a-9dc8-6d3eab1c7933. >2014-08-11 00:37:53,404 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-73) Principal admin::internal synchronized >2014-08-11 00:37:58,112 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-86) Attempting to update VMs/Templates Ovf. >2014-08-11 00:42:55,812 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-11) START, GetFileStatsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 1bed9648 >2014-08-11 00:42:55,828 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-11) FINISH, GetFileStatsVDSCommand, return: {test.iso={status=0, ctime=1407430000.0, size=4}}, log id: 1bed9648 >2014-08-11 00:42:55,833 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (org.ovirt.thread.pool-8-thread-11) Finished automatic refresh process for ISO file type with success, for storage domain id 63fe3516-4170-402a-9dc8-6d3eab1c7933. >2014-08-11 01:37:53,414 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-43) Principal admin::internal synchronized >2014-08-11 01:37:58,137 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-79) Attempting to update VMs/Templates Ovf. >2014-08-11 01:42:55,843 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-11) START, GetFileStatsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 4f9e4a7 >2014-08-11 01:42:55,860 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-11) FINISH, GetFileStatsVDSCommand, return: {test.iso={status=0, ctime=1407430000.0, size=4}}, log id: 4f9e4a7 >2014-08-11 01:42:55,864 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (org.ovirt.thread.pool-8-thread-11) Finished automatic refresh process for ISO file type with success, for storage domain id 63fe3516-4170-402a-9dc8-6d3eab1c7933. >2014-08-11 02:37:53,420 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-15) [6fcc1be5] Principal admin::internal synchronized >2014-08-11 02:37:58,172 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-52) Attempting to update VMs/Templates Ovf. >2014-08-11 02:42:55,877 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-11) START, GetFileStatsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 7bc89122 >2014-08-11 02:42:55,894 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-11) FINISH, GetFileStatsVDSCommand, return: {test.iso={status=0, ctime=1407430000.0, size=4}}, log id: 7bc89122 >2014-08-11 02:42:55,900 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (org.ovirt.thread.pool-8-thread-11) Finished automatic refresh process for ISO file type with success, for storage domain id 63fe3516-4170-402a-9dc8-6d3eab1c7933. >2014-08-11 03:35:35,001 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-84) [7f085c60] Start deleteAgedOutAuditLogs >2014-08-11 03:35:35,002 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-17) Start deleteAgedOutCommandEntities >2014-08-11 03:35:35,010 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-84) [7f085c60] Finished deleteAgedOutAuditLogs >2014-08-11 03:35:35,011 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-17) Finished deleteAgedOutCommandEntities >2014-08-11 03:37:53,428 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-33) Principal admin::internal synchronized >2014-08-11 03:37:58,195 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-65) Attempting to update VMs/Templates Ovf. >2014-08-11 03:42:55,907 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-16) START, GetFileStatsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 2f02881d >2014-08-11 03:42:55,923 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-16) FINISH, GetFileStatsVDSCommand, return: {test.iso={status=0, ctime=1407430000.0, size=4}}, log id: 2f02881d >2014-08-11 03:42:55,929 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (org.ovirt.thread.pool-8-thread-16) Finished automatic refresh process for ISO file type with success, for storage domain id 63fe3516-4170-402a-9dc8-6d3eab1c7933. >2014-08-11 04:37:53,437 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-71) Principal admin::internal synchronized >2014-08-11 04:37:58,215 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-99) Attempting to update VMs/Templates Ovf. >2014-08-11 04:42:55,938 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-18) START, GetFileStatsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 12beff5c >2014-08-11 04:42:55,955 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-18) FINISH, GetFileStatsVDSCommand, return: {test.iso={status=0, ctime=1407430000.0, size=4}}, log id: 12beff5c >2014-08-11 04:42:55,960 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (org.ovirt.thread.pool-8-thread-18) Finished automatic refresh process for ISO file type with success, for storage domain id 63fe3516-4170-402a-9dc8-6d3eab1c7933. >2014-08-11 05:37:53,444 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-16) Principal admin::internal synchronized >2014-08-11 05:37:58,250 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-98) Attempting to update VMs/Templates Ovf. >2014-08-11 05:42:55,973 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-19) START, GetFileStatsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 41893219 >2014-08-11 05:42:55,990 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-19) FINISH, GetFileStatsVDSCommand, return: {test.iso={status=0, ctime=1407430000.0, size=4}}, log id: 41893219 >2014-08-11 05:42:55,996 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (org.ovirt.thread.pool-8-thread-19) Finished automatic refresh process for ISO file type with success, for storage domain id 63fe3516-4170-402a-9dc8-6d3eab1c7933. >2014-08-11 06:37:53,450 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-34) Principal admin::internal synchronized >2014-08-11 06:37:58,287 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-92) Attempting to update VMs/Templates Ovf. >2014-08-11 06:42:56,005 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-24) START, GetFileStatsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 429a88fa >2014-08-11 06:42:56,020 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-24) FINISH, GetFileStatsVDSCommand, return: {test.iso={status=0, ctime=1407430000.0, size=4}}, log id: 429a88fa >2014-08-11 06:42:56,026 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (org.ovirt.thread.pool-8-thread-24) Finished automatic refresh process for ISO file type with success, for storage domain id 63fe3516-4170-402a-9dc8-6d3eab1c7933. >2014-08-11 07:37:53,455 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-37) Principal admin::internal synchronized >2014-08-11 07:37:58,309 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-40) Attempting to update VMs/Templates Ovf. >2014-08-11 07:42:56,034 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-44) START, GetFileStatsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 3f3981e >2014-08-11 07:42:56,050 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-44) FINISH, GetFileStatsVDSCommand, return: {test.iso={status=0, ctime=1407430000.0, size=4}}, log id: 3f3981e >2014-08-11 07:42:56,055 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (org.ovirt.thread.pool-8-thread-44) Finished automatic refresh process for ISO file type with success, for storage domain id 63fe3516-4170-402a-9dc8-6d3eab1c7933. >2014-08-11 08:37:53,461 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-24) Principal admin::internal synchronized >2014-08-11 08:37:58,328 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-62) Attempting to update VMs/Templates Ovf. >2014-08-11 08:42:56,065 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-12) START, GetFileStatsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 39df1572 >2014-08-11 08:42:56,082 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-12) FINISH, GetFileStatsVDSCommand, return: {test.iso={status=0, ctime=1407430000.0, size=4}}, log id: 39df1572 >2014-08-11 08:42:56,088 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (org.ovirt.thread.pool-8-thread-12) Finished automatic refresh process for ISO file type with success, for storage domain id 63fe3516-4170-402a-9dc8-6d3eab1c7933. >2014-08-11 08:50:39,064 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-5) Running command: LoginUserCommand internal: false. >2014-08-11 08:50:39,072 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-5) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-08-11 08:50:48,689 INFO [org.ovirt.engine.core.bll.aaa.LoginAdminUserCommand] (ajp--127.0.0.1-8702-1) Running command: LoginAdminUserCommand internal: false. >2014-08-11 08:50:48,694 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-1) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-08-11 09:02:44,116 ERROR [org.ovirt.engine.ui.frontend.server.gwt.GenericApiGWTServiceImpl] (ajp--127.0.0.1-8702-2) Retrieving non string value from session >2014-08-11 09:36:26,701 WARN [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-8-thread-27) Executing a command: java.util.concurrent.FutureTask , but note that there are 1 tasks in the queue. >2014-08-11 09:37:53,470 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-14) Principal admin::internal synchronized >2014-08-11 09:37:58,366 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-7) Attempting to update VMs/Templates Ovf. >2014-08-11 09:42:56,097 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-8) [6fcc1be5] START, GetFileStatsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 1a385435 >2014-08-11 09:42:56,121 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-8) [6fcc1be5] FINISH, GetFileStatsVDSCommand, return: {test.iso={status=0, ctime=1407430000.0, size=4}}, log id: 1a385435 >2014-08-11 09:42:56,130 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (org.ovirt.thread.pool-8-thread-8) [6fcc1be5] Finished automatic refresh process for ISO file type with success, for storage domain id 63fe3516-4170-402a-9dc8-6d3eab1c7933. >2014-08-11 10:37:53,478 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-26) Principal admin::internal synchronized >2014-08-11 10:37:58,406 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-17) Attempting to update VMs/Templates Ovf. >2014-08-11 10:42:56,142 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-26) START, GetFileStatsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 352e7fdd >2014-08-11 10:42:56,159 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-26) FINISH, GetFileStatsVDSCommand, return: {test.iso={status=0, ctime=1407430000.0, size=4}}, log id: 352e7fdd >2014-08-11 10:42:56,165 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (org.ovirt.thread.pool-8-thread-26) Finished automatic refresh process for ISO file type with success, for storage domain id 63fe3516-4170-402a-9dc8-6d3eab1c7933. >2014-08-11 11:37:53,488 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-56) Principal admin::internal synchronized >2014-08-11 11:37:58,441 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-42) Attempting to update VMs/Templates Ovf. >2014-08-11 11:42:56,174 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-9) START, GetFileStatsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 28eaa2b6 >2014-08-11 11:42:56,192 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-9) FINISH, GetFileStatsVDSCommand, return: {test.iso={status=0, ctime=1407430000.0, size=4}}, log id: 28eaa2b6 >2014-08-11 11:42:56,197 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (org.ovirt.thread.pool-8-thread-9) Finished automatic refresh process for ISO file type with success, for storage domain id 63fe3516-4170-402a-9dc8-6d3eab1c7933. >2014-08-11 12:37:53,495 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-89) Principal admin::internal synchronized >2014-08-11 12:37:58,486 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-16) Attempting to update VMs/Templates Ovf. >2014-08-11 12:42:56,204 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-7) START, GetFileStatsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 3ea5c47a >2014-08-11 12:42:56,220 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-7) FINISH, GetFileStatsVDSCommand, return: {test.iso={status=0, ctime=1407430000.0, size=4}}, log id: 3ea5c47a >2014-08-11 12:42:56,225 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (org.ovirt.thread.pool-8-thread-7) Finished automatic refresh process for ISO file type with success, for storage domain id 63fe3516-4170-402a-9dc8-6d3eab1c7933. >2014-08-11 13:05:30,244 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-7) Running command: LoginUserCommand internal: false. >2014-08-11 13:05:30,256 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-7) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. >2014-08-11 13:05:30,266 INFO [org.ovirt.engine.core.bll.aaa.LogoutBySessionCommand] (ajp--127.0.0.1-8702-7) [13e65a3e] Running command: LogoutBySessionCommand internal: false. >2014-08-11 13:05:30,270 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp--127.0.0.1-8702-7) [489031bf] Running command: LogoutUserCommand internal: false. >2014-08-11 13:05:30,279 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-7) [489031bf] Correlation ID: 489031bf, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-08-11 13:37:53,503 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-18) Principal admin::internal synchronized >2014-08-11 13:37:58,505 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-81) Attempting to update VMs/Templates Ovf. >2014-08-11 13:42:56,238 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-1) [175991ea] START, GetFileStatsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 4ba7ee0d >2014-08-11 13:42:56,254 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-1) [175991ea] FINISH, GetFileStatsVDSCommand, return: {test.iso={status=0, ctime=1407430000.0, size=4}}, log id: 4ba7ee0d >2014-08-11 13:42:56,260 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (org.ovirt.thread.pool-8-thread-1) [175991ea] Finished automatic refresh process for ISO file type with success, for storage domain id 63fe3516-4170-402a-9dc8-6d3eab1c7933. >2014-08-11 14:04:41,968 ERROR [org.ovirt.engine.ui.frontend.server.gwt.GenericApiGWTServiceImpl] (ajp--127.0.0.1-8702-8) Retrieving non string value from session >2014-08-11 14:37:53,513 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-37) Principal admin::internal synchronized >2014-08-11 14:37:58,525 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-59) Attempting to update VMs/Templates Ovf. >2014-08-11 14:42:56,272 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-34) START, GetFileStatsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 4aa2e265 >2014-08-11 14:42:56,289 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-34) FINISH, GetFileStatsVDSCommand, return: {test.iso={status=0, ctime=1407430000.0, size=4}}, log id: 4aa2e265 >2014-08-11 14:42:56,296 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (org.ovirt.thread.pool-8-thread-34) Finished automatic refresh process for ISO file type with success, for storage domain id 63fe3516-4170-402a-9dc8-6d3eab1c7933. >2014-08-11 15:37:53,519 INFO [org.ovirt.engine.core.bll.aaa.SyncUsers] (DefaultQuartzScheduler_Worker-10) Principal admin::internal synchronized >2014-08-11 15:37:58,548 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-24) Attempting to update VMs/Templates Ovf. >2014-08-11 15:42:56,308 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-30) START, GetFileStatsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false), log id: 19730428 >2014-08-11 15:42:56,324 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-30) FINISH, GetFileStatsVDSCommand, return: {test.iso={status=0, ctime=1407430000.0, size=4}}, log id: 19730428 >2014-08-11 15:42:56,329 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (org.ovirt.thread.pool-8-thread-30) Finished automatic refresh process for ISO file type with success, for storage domain id 63fe3516-4170-402a-9dc8-6d3eab1c7933. >2014-08-11 15:57:22,598 WARN [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-8-thread-44) Executing a command: java.util.concurrent.FutureTask , but note that there are 1 tasks in the queue. >2014-08-11 16:00:56,593 ERROR [org.ovirt.engine.ui.frontend.server.gwt.GenericApiGWTServiceImpl] (ajp--127.0.0.1-8702-4) Retrieving non string value from session >2014-08-11 16:01:07,107 ERROR [org.ovirt.engine.ui.frontend.server.gwt.GenericApiGWTServiceImpl] (ajp--127.0.0.1-8702-8) Retrieving non string value from session >2014-08-11 16:02:17,868 INFO [org.ovirt.engine.core.bll.MaintenanceNumberOfVdssCommand] (ajp--127.0.0.1-8702-2) [6288851b] Lock Acquired to object EngineLock [exclusiveLocks= , sharedLocks= key: 00000002-0002-0002-0002-0000000001d9 value: POOL >] >2014-08-11 16:02:17,910 INFO [org.ovirt.engine.core.bll.MaintenanceNumberOfVdssCommand] (org.ovirt.thread.pool-8-thread-39) [6288851b] Running command: MaintenanceNumberOfVdssCommand internal: false. Entities affected : ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS >2014-08-11 16:02:17,917 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-8-thread-39) [6288851b] START, SetVdsStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, status=PreparingForMaintenance, nonOperationalReason=NONE, stopSpmFailureLogged=true), log id: d935e35 >2014-08-11 16:02:17,918 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-8-thread-39) [6288851b] VDS rack is spm and moved from up calling resetIrs. >2014-08-11 16:02:17,920 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-39) [6288851b] START, ResetIrsVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, ignoreStopFailed = false), log id: 5ebbdd55 >2014-08-11 16:02:17,926 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-39) [6288851b] START, SpmStopVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9), log id: 4bfb1b8 >2014-08-11 16:02:17,932 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-39) [6288851b] SpmStopVDSCommand::Stopping SPM on vds rack, pool id 00000002-0002-0002-0002-0000000001d9 >2014-08-11 16:02:17,988 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-39) [6288851b] FINISH, SpmStopVDSCommand, log id: 4bfb1b8 >2014-08-11 16:02:17,996 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-39) [6288851b] FINISH, ResetIrsVDSCommand, log id: 5ebbdd55 >2014-08-11 16:02:18,002 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-8-thread-39) [6288851b] FINISH, SetVdsStatusVDSCommand, log id: d935e35 >2014-08-11 16:02:18,057 INFO [org.ovirt.engine.core.bll.MaintenanceNumberOfVdssCommand] (org.ovirt.thread.pool-8-thread-39) [6288851b] Lock freed to object EngineLock [exclusiveLocks= , sharedLocks= key: 00000002-0002-0002-0002-0000000001d9 value: POOL >] >2014-08-11 16:02:18,089 INFO [org.ovirt.engine.core.bll.MaintenanceVdsCommand] (org.ovirt.thread.pool-8-thread-39) [6288851b] Running command: MaintenanceVdsCommand internal: true. Entities affected : ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS >2014-08-11 16:02:18,133 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-39) [6288851b] Correlation ID: 6288851b, Job ID: c366ecce-d4db-4b00-9106-45a2f0998c3e, Call Stack: null, Custom Event ID: -1, Message: Host rack was switched to Maintenance mode by admin. >2014-08-11 16:02:19,932 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-32) Updated vds status from Preparing for Maintenance to Maintenance in database, vds = 17f7fe56-5f32-40ef-b098-611b84484e0b : rack >2014-08-11 16:02:19,961 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-49) Clearing cache of pool: 00000002-0002-0002-0002-0000000001d9 for problematic entities of VDS: rack. >2014-08-11 16:02:19,962 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-49) Removing vds [17f7fe56-5f32-40ef-b098-611b84484e0b] from the domain in maintenance cache >2014-08-11 16:02:19,964 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-49) Removing host(s) [17f7fe56-5f32-40ef-b098-611b84484e0b] from hosts unseen domain report cache >2014-08-11 16:02:19,969 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-32) START, DisconnectStoragePoolVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, vds_spm_id = 1), log id: cdfd24d >2014-08-11 16:02:22,444 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-32) FINISH, DisconnectStoragePoolVDSCommand, log id: cdfd24d >2014-08-11 16:02:22,451 INFO [org.ovirt.engine.core.bll.storage.DisconnectHostFromStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-32) [54f2a865] Running command: DisconnectHostFromStoragePoolServersCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-08-11 16:02:22,471 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-32) [54f2a865] START, DisconnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, storageType = NFS, connectionList = [{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 7fe036a2-2dfc-4565-b936-69a678cf0613, connection: pb-rh35.rhev.lab.eng.brq.redhat.com:/var/lib/exports/iso, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 26390da9 >2014-08-11 16:02:22,841 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-32) [54f2a865] FINISH, DisconnectStorageServerVDSCommand, return: {7fe036a2-2dfc-4565-b936-69a678cf0613=0, 5083f370-af09-457b-9b73-3440e8b65a07=0}, log id: 26390da9 >2014-08-11 16:02:25,368 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-58) Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 63fe3516-4170-402a-9dc8-6d3eab1c7933 status from Active to Unknown, reason : no reporting hosts >2014-08-11 16:02:25,372 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-58) Storage Pool 00000002-0002-0002-0002-0000000001d9 - Updating Storage Domain 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 status from Active to Unknown, reason : no reporting hosts >2014-08-11 16:02:25,378 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-58) [13e2eaf6] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-08-11 16:02:25,389 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-58) [13e2eaf6] Correlation ID: 13e2eaf6, Call Stack: null, Custom Event ID: -1, Message: Invalid status on Data Center Default. Setting status to Non Responsive. >2014-08-11 16:02:30,570 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-7) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-11 16:02:30,571 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-7) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-11 16:02:30,573 INFO [org.ovirt.engine.core.bll.UpdateVdsCommand] (ajp--127.0.0.1-8702-7) [71c95186] Running command: UpdateVdsCommand internal: false. Entities affected : ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS >2014-08-11 16:02:30,614 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-7) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-11 16:02:30,615 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp--127.0.0.1-8702-7) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-08-11 16:02:30,622 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-7) [71c95186] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host rack. >2014-08-11 16:02:30,639 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-7) [71c95186] Correlation ID: 71c95186, Call Stack: null, Custom Event ID: -1, Message: Host rack configuration was updated by admin. >2014-08-11 16:02:34,897 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (ajp--127.0.0.1-8702-7) [287267fa] Lock Acquired to object EngineLock [exclusiveLocks= key: 17f7fe56-5f32-40ef-b098-611b84484e0b value: VDS >, sharedLocks= ] >2014-08-11 16:02:34,931 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (org.ovirt.thread.pool-8-thread-41) [287267fa] Running command: ActivateVdsCommand internal: false. Entities affected : ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS >2014-08-11 16:02:34,934 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (org.ovirt.thread.pool-8-thread-41) [287267fa] Before acquiring lock in order to prevent monitoring for host rack from data-center Default >2014-08-11 16:02:34,935 INFO [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (org.ovirt.thread.pool-8-thread-41) [287267fa] Failed to acquire lock and wait lock EngineLock [exclusiveLocks= key: 17f7fe56-5f32-40ef-b098-611b84484e0b value: VDS_INIT >, sharedLocks= ] >2014-08-11 16:02:34,948 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (org.ovirt.thread.pool-8-thread-41) [287267fa] Lock acquired, from now a monitoring of host will be skipped for host rack from data-center Default >2014-08-11 16:02:34,966 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-8-thread-41) [287267fa] START, SetVdsStatusVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 721eb9fe >2014-08-11 16:02:34,973 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-8-thread-41) [287267fa] FINISH, SetVdsStatusVDSCommand, log id: 721eb9fe >2014-08-11 16:02:35,013 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (org.ovirt.thread.pool-8-thread-41) [287267fa] Activate finished. Lock released. Monitoring can run now for host rack from data-center Default >2014-08-11 16:02:35,021 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-41) [287267fa] Correlation ID: 287267fa, Job ID: 0a945613-5f69-4fb9-8f1f-0fa218caf432, Call Stack: null, Custom Event ID: -1, Message: Host rack was activated by admin. >2014-08-11 16:02:35,027 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (org.ovirt.thread.pool-8-thread-41) [287267fa] Lock freed to object EngineLock [exclusiveLocks= key: 17f7fe56-5f32-40ef-b098-611b84484e0b value: VDS >, sharedLocks= ] >2014-08-11 16:02:38,055 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-81) START, GetHardwareInfoVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vds=Host[rack,17f7fe56-5f32-40ef-b098-611b84484e0b]), log id: 1b79d63f >2014-08-11 16:02:38,106 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-81) FINISH, GetHardwareInfoVDSCommand, log id: 1b79d63f >2014-08-11 16:02:38,134 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-81) [33c8cfcd] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS >2014-08-11 16:02:38,186 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-81) [72966734] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-08-11 16:02:38,198 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-81) [229df7c9] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-08-11 16:02:38,214 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-81) [229df7c9] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, storageType = NFS, connectionList = [{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 7fe036a2-2dfc-4565-b936-69a678cf0613, connection: pb-rh35.rhev.lab.eng.brq.redhat.com:/var/lib/exports/iso, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 60859deb >2014-08-11 16:02:43,607 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-81) [229df7c9] FINISH, ConnectStorageServerVDSCommand, return: {7fe036a2-2dfc-4565-b936-69a678cf0613=0, 5083f370-af09-457b-9b73-3440e8b65a07=0}, log id: 60859deb >2014-08-11 16:02:43,609 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-81) [229df7c9] Host rack storage connection was succeeded >2014-08-11 16:02:43,618 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-30) START, ConnectStoragePoolVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, vdsId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, masterVersion = 1), log id: 1c03780 >2014-08-11 16:02:45,119 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-30) FINISH, ConnectStoragePoolVDSCommand, log id: 1c03780 >2014-08-11 16:02:45,133 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-81) [229df7c9] START, SetMOMPolicyParametersVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b), log id: 70c0d85e >2014-08-11 16:02:45,230 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-81) [229df7c9] FINISH, SetMOMPolicyParametersVDSCommand, log id: 70c0d85e >2014-08-11 16:02:45,239 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-81) [229df7c9] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host rack was set to Up. >2014-08-11 16:02:45,245 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-81) [229df7c9] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host rack. >2014-08-11 16:02:45,268 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-81) [4e9c0ea5] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17f7fe56-5f32-40ef-b098-611b84484e0b Type: VDS >2014-08-11 16:02:45,275 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-31) Adding domain 44559ec0-6bef-491c-bbd6-e67278f37987 to the domains in maintenance cache >2014-08-11 16:02:45,546 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-42) hostFromVds::selectedVds - rack, spmStatus Free, storage pool Default >2014-08-11 16:02:45,552 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-42) starting spm on vds rack, storage pool Default, prevId -1, LVER -1 >2014-08-11 16:02:45,555 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-42) START, SpmStartVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000002-0002-0002-0002-0000000001d9, prevId=-1, prevLVER=-1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 71d49d48 >2014-08-11 16:02:45,603 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-42) spmStart polling started: taskId = c33e7f3a-9e54-4d1b-9431-a5eded1a66f6 >2014-08-11 16:03:07,746 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-42) spmStart polling ended: taskId = c33e7f3a-9e54-4d1b-9431-a5eded1a66f6 task status = finished >2014-08-11 16:03:07,802 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-42) spmStart polling ended, spm status: SPM >2014-08-11 16:03:07,805 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-42) START, HSMClearTaskVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, taskId=c33e7f3a-9e54-4d1b-9431-a5eded1a66f6), log id: 66f3dd52 >2014-08-11 16:03:07,849 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-42) FINISH, HSMClearTaskVDSCommand, log id: 66f3dd52 >2014-08-11 16:03:07,854 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-42) FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@1094465d, log id: 71d49d48 >2014-08-11 16:03:07,862 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-42) Initialize Irs proxy from vds: 10.34.62.208 >2014-08-11 16:03:07,868 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-42) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208). >2014-08-11 16:03:07,907 ERROR [org.ovirt.engine.core.utils.timer.SchedulerUtilQuartzImpl] (DefaultQuartzScheduler_Worker-42) Failed to invoke scheduled method _updatingTimer_Elapsed: java.lang.reflect.InvocationTargetException > at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source) [:1.7.0_55] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.lang.NoSuchMethodError: org.ovirt.vdsm.jsonrpc.client.utils.retry.RetryPolicy.<init>(IIILjava/lang/Class;)V > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.getJsonClient(JsonRpcUtils.java:48) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.createClient(JsonRpcUtils.java:38) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.createStompClient(JsonRpcUtils.java:22) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.TransportFactory.createIrsServer(TransportFactory.java:21) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData.getIrsProxy(IrsProxyData.java:579) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand.getIrsProxy(IrsBrokerCommand.java:146) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand.executeVDSCommand(IrsBrokerCommand.java:154) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:407) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData.proceedStoragePoolStats(IrsProxyData.java:293) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData._updatingTimer_Elapsed(IrsProxyData.java:187) [vdsbroker.jar:] > ... 6 more > >2014-08-11 16:03:17,960 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-22) [1144b5b] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-08-11 16:03:17,972 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-22) [1144b5b] Correlation ID: 1144b5b, Call Stack: null, Custom Event ID: -1, Message: Data Center is being initialized, please wait for initialization to complete. >2014-08-11 16:03:18,030 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-22) [1144b5b] hostFromVds::selectedVds - rack, spmStatus SPM, storage pool Default >2014-08-11 16:03:18,033 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-22) [1144b5b] Initialize Irs proxy from vds: 10.34.62.208 >2014-08-11 16:03:18,041 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-22) [1144b5b] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208). >2014-08-11 16:03:18,042 ERROR [org.ovirt.engine.core.utils.timer.SchedulerUtilQuartzImpl] (DefaultQuartzScheduler_Worker-22) Failed to invoke scheduled method _updatingTimer_Elapsed: java.lang.reflect.InvocationTargetException > at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source) [:1.7.0_55] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.lang.NoSuchMethodError: org.ovirt.vdsm.jsonrpc.client.utils.retry.RetryPolicy.<init>(IIILjava/lang/Class;)V > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.getJsonClient(JsonRpcUtils.java:48) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.createClient(JsonRpcUtils.java:38) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.createStompClient(JsonRpcUtils.java:22) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.TransportFactory.createIrsServer(TransportFactory.java:21) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData.getIrsProxy(IrsProxyData.java:579) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand.getIrsProxy(IrsBrokerCommand.java:146) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand.executeVDSCommand(IrsBrokerCommand.java:154) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:407) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData.proceedStoragePoolStats(IrsProxyData.java:293) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData._updatingTimer_Elapsed(IrsProxyData.java:187) [vdsbroker.jar:] > ... 6 more > >2014-08-11 16:03:28,095 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-91) [35fa9ee2] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-08-11 16:03:28,161 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-91) [35fa9ee2] hostFromVds::selectedVds - rack, spmStatus SPM, storage pool Default >2014-08-11 16:03:28,169 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-91) [35fa9ee2] Initialize Irs proxy from vds: 10.34.62.208 >2014-08-11 16:03:28,180 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-91) [35fa9ee2] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208). >2014-08-11 16:03:28,183 ERROR [org.ovirt.engine.core.utils.timer.SchedulerUtilQuartzImpl] (DefaultQuartzScheduler_Worker-91) Failed to invoke scheduled method _updatingTimer_Elapsed: java.lang.reflect.InvocationTargetException > at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source) [:1.7.0_55] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.lang.NoSuchMethodError: org.ovirt.vdsm.jsonrpc.client.utils.retry.RetryPolicy.<init>(IIILjava/lang/Class;)V > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.getJsonClient(JsonRpcUtils.java:48) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.createClient(JsonRpcUtils.java:38) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.createStompClient(JsonRpcUtils.java:22) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.TransportFactory.createIrsServer(TransportFactory.java:21) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData.getIrsProxy(IrsProxyData.java:579) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand.getIrsProxy(IrsBrokerCommand.java:146) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand.executeVDSCommand(IrsBrokerCommand.java:154) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:407) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData.proceedStoragePoolStats(IrsProxyData.java:293) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData._updatingTimer_Elapsed(IrsProxyData.java:187) [vdsbroker.jar:] > ... 6 more > >2014-08-11 16:03:38,300 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-34) [67bcf03e] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-08-11 16:03:38,433 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-34) [67bcf03e] hostFromVds::selectedVds - rack, spmStatus SPM, storage pool Default >2014-08-11 16:03:38,436 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-34) [67bcf03e] Initialize Irs proxy from vds: 10.34.62.208 >2014-08-11 16:03:38,443 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-34) [67bcf03e] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208). >2014-08-11 16:03:38,444 ERROR [org.ovirt.engine.core.utils.timer.SchedulerUtilQuartzImpl] (DefaultQuartzScheduler_Worker-34) Failed to invoke scheduled method _updatingTimer_Elapsed: java.lang.reflect.InvocationTargetException > at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source) [:1.7.0_55] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.lang.NoSuchMethodError: org.ovirt.vdsm.jsonrpc.client.utils.retry.RetryPolicy.<init>(IIILjava/lang/Class;)V > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.getJsonClient(JsonRpcUtils.java:48) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.createClient(JsonRpcUtils.java:38) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.createStompClient(JsonRpcUtils.java:22) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.TransportFactory.createIrsServer(TransportFactory.java:21) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData.getIrsProxy(IrsProxyData.java:579) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand.getIrsProxy(IrsBrokerCommand.java:146) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand.executeVDSCommand(IrsBrokerCommand.java:154) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:407) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData.proceedStoragePoolStats(IrsProxyData.java:293) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData._updatingTimer_Elapsed(IrsProxyData.java:187) [vdsbroker.jar:] > ... 6 more > >2014-08-11 16:03:48,500 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-50) [6d88073] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-08-11 16:03:48,565 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-50) [6d88073] hostFromVds::selectedVds - rack, spmStatus SPM, storage pool Default >2014-08-11 16:03:48,569 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-50) [6d88073] Initialize Irs proxy from vds: 10.34.62.208 >2014-08-11 16:03:48,575 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-50) [6d88073] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208). >2014-08-11 16:03:48,577 ERROR [org.ovirt.engine.core.utils.timer.SchedulerUtilQuartzImpl] (DefaultQuartzScheduler_Worker-50) Failed to invoke scheduled method _updatingTimer_Elapsed: java.lang.reflect.InvocationTargetException > at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source) [:1.7.0_55] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.lang.NoSuchMethodError: org.ovirt.vdsm.jsonrpc.client.utils.retry.RetryPolicy.<init>(IIILjava/lang/Class;)V > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.getJsonClient(JsonRpcUtils.java:48) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.createClient(JsonRpcUtils.java:38) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.createStompClient(JsonRpcUtils.java:22) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.TransportFactory.createIrsServer(TransportFactory.java:21) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData.getIrsProxy(IrsProxyData.java:579) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand.getIrsProxy(IrsBrokerCommand.java:146) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand.executeVDSCommand(IrsBrokerCommand.java:154) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:407) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData.proceedStoragePoolStats(IrsProxyData.java:293) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData._updatingTimer_Elapsed(IrsProxyData.java:187) [vdsbroker.jar:] > ... 6 more > >2014-08-11 16:03:58,630 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-48) [53a642c3] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-08-11 16:03:58,696 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-48) [53a642c3] hostFromVds::selectedVds - rack, spmStatus SPM, storage pool Default >2014-08-11 16:03:58,699 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-48) [53a642c3] Initialize Irs proxy from vds: 10.34.62.208 >2014-08-11 16:03:58,705 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-48) [53a642c3] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208). >2014-08-11 16:03:58,707 ERROR [org.ovirt.engine.core.utils.timer.SchedulerUtilQuartzImpl] (DefaultQuartzScheduler_Worker-48) Failed to invoke scheduled method _updatingTimer_Elapsed: java.lang.reflect.InvocationTargetException > at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source) [:1.7.0_55] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.lang.NoSuchMethodError: org.ovirt.vdsm.jsonrpc.client.utils.retry.RetryPolicy.<init>(IIILjava/lang/Class;)V > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.getJsonClient(JsonRpcUtils.java:48) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.createClient(JsonRpcUtils.java:38) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.createStompClient(JsonRpcUtils.java:22) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.TransportFactory.createIrsServer(TransportFactory.java:21) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData.getIrsProxy(IrsProxyData.java:579) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand.getIrsProxy(IrsBrokerCommand.java:146) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand.executeVDSCommand(IrsBrokerCommand.java:154) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:407) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData.proceedStoragePoolStats(IrsProxyData.java:293) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData._updatingTimer_Elapsed(IrsProxyData.java:187) [vdsbroker.jar:] > ... 6 more > >2014-08-11 16:04:08,761 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-23) [8bd4403] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-08-11 16:04:08,827 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-23) [8bd4403] hostFromVds::selectedVds - rack, spmStatus SPM, storage pool Default >2014-08-11 16:04:08,831 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-23) [8bd4403] Initialize Irs proxy from vds: 10.34.62.208 >2014-08-11 16:04:08,837 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-23) [8bd4403] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208). >2014-08-11 16:04:08,838 ERROR [org.ovirt.engine.core.utils.timer.SchedulerUtilQuartzImpl] (DefaultQuartzScheduler_Worker-23) Failed to invoke scheduled method _updatingTimer_Elapsed: java.lang.reflect.InvocationTargetException > at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source) [:1.7.0_55] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.lang.NoSuchMethodError: org.ovirt.vdsm.jsonrpc.client.utils.retry.RetryPolicy.<init>(IIILjava/lang/Class;)V > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.getJsonClient(JsonRpcUtils.java:48) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.createClient(JsonRpcUtils.java:38) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.createStompClient(JsonRpcUtils.java:22) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.TransportFactory.createIrsServer(TransportFactory.java:21) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData.getIrsProxy(IrsProxyData.java:579) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand.getIrsProxy(IrsBrokerCommand.java:146) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand.executeVDSCommand(IrsBrokerCommand.java:154) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:407) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData.proceedStoragePoolStats(IrsProxyData.java:293) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData._updatingTimer_Elapsed(IrsProxyData.java:187) [vdsbroker.jar:] > ... 6 more > >2014-08-11 16:04:18,910 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-82) [2249f537] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-08-11 16:04:18,936 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-82) [2249f537] Correlation ID: 2249f537, Call Stack: null, Custom Event ID: -1, Message: Data Center is being initialized, please wait for initialization to complete. >2014-08-11 16:04:18,960 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-82) [2249f537] hostFromVds::selectedVds - rack, spmStatus SPM, storage pool Default >2014-08-11 16:04:18,966 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-82) [2249f537] Initialize Irs proxy from vds: 10.34.62.208 >2014-08-11 16:04:18,974 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-82) [2249f537] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208). >2014-08-11 16:04:18,976 ERROR [org.ovirt.engine.core.utils.timer.SchedulerUtilQuartzImpl] (DefaultQuartzScheduler_Worker-82) Failed to invoke scheduled method _updatingTimer_Elapsed: java.lang.reflect.InvocationTargetException > at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source) [:1.7.0_55] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.lang.NoSuchMethodError: org.ovirt.vdsm.jsonrpc.client.utils.retry.RetryPolicy.<init>(IIILjava/lang/Class;)V > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.getJsonClient(JsonRpcUtils.java:48) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.createClient(JsonRpcUtils.java:38) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.createStompClient(JsonRpcUtils.java:22) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.TransportFactory.createIrsServer(TransportFactory.java:21) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData.getIrsProxy(IrsProxyData.java:579) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand.getIrsProxy(IrsBrokerCommand.java:146) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand.executeVDSCommand(IrsBrokerCommand.java:154) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:407) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData.proceedStoragePoolStats(IrsProxyData.java:293) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData._updatingTimer_Elapsed(IrsProxyData.java:187) [vdsbroker.jar:] > ... 6 more > >2014-08-11 16:04:29,038 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-89) [497a3d78] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-08-11 16:04:29,104 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-89) [497a3d78] hostFromVds::selectedVds - rack, spmStatus SPM, storage pool Default >2014-08-11 16:04:29,108 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-89) [497a3d78] Initialize Irs proxy from vds: 10.34.62.208 >2014-08-11 16:04:29,115 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-89) [497a3d78] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208). >2014-08-11 16:04:29,117 ERROR [org.ovirt.engine.core.utils.timer.SchedulerUtilQuartzImpl] (DefaultQuartzScheduler_Worker-89) Failed to invoke scheduled method _updatingTimer_Elapsed: java.lang.reflect.InvocationTargetException > at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source) [:1.7.0_55] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.lang.NoSuchMethodError: org.ovirt.vdsm.jsonrpc.client.utils.retry.RetryPolicy.<init>(IIILjava/lang/Class;)V > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.getJsonClient(JsonRpcUtils.java:48) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.createClient(JsonRpcUtils.java:38) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.createStompClient(JsonRpcUtils.java:22) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.TransportFactory.createIrsServer(TransportFactory.java:21) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData.getIrsProxy(IrsProxyData.java:579) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand.getIrsProxy(IrsBrokerCommand.java:146) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand.executeVDSCommand(IrsBrokerCommand.java:154) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:407) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData.proceedStoragePoolStats(IrsProxyData.java:293) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData._updatingTimer_Elapsed(IrsProxyData.java:187) [vdsbroker.jar:] > ... 6 more > >2014-08-11 16:04:39,195 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-54) [996c68a] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-08-11 16:04:39,260 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-54) [996c68a] hostFromVds::selectedVds - rack, spmStatus SPM, storage pool Default >2014-08-11 16:04:39,263 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-54) [996c68a] Initialize Irs proxy from vds: 10.34.62.208 >2014-08-11 16:04:39,270 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-54) [996c68a] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208). >2014-08-11 16:04:39,272 ERROR [org.ovirt.engine.core.utils.timer.SchedulerUtilQuartzImpl] (DefaultQuartzScheduler_Worker-54) Failed to invoke scheduled method _updatingTimer_Elapsed: java.lang.reflect.InvocationTargetException > at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source) [:1.7.0_55] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.lang.NoSuchMethodError: org.ovirt.vdsm.jsonrpc.client.utils.retry.RetryPolicy.<init>(IIILjava/lang/Class;)V > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.getJsonClient(JsonRpcUtils.java:48) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.createClient(JsonRpcUtils.java:38) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.createStompClient(JsonRpcUtils.java:22) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.TransportFactory.createIrsServer(TransportFactory.java:21) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData.getIrsProxy(IrsProxyData.java:579) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand.getIrsProxy(IrsBrokerCommand.java:146) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand.executeVDSCommand(IrsBrokerCommand.java:154) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:407) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData.proceedStoragePoolStats(IrsProxyData.java:293) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData._updatingTimer_Elapsed(IrsProxyData.java:187) [vdsbroker.jar:] > ... 6 more > >2014-08-11 16:04:49,329 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-45) [7cf5d75e] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-08-11 16:04:49,394 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-45) [7cf5d75e] hostFromVds::selectedVds - rack, spmStatus SPM, storage pool Default >2014-08-11 16:04:49,398 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-45) [7cf5d75e] Initialize Irs proxy from vds: 10.34.62.208 >2014-08-11 16:04:49,406 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-45) [7cf5d75e] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208). >2014-08-11 16:04:49,407 ERROR [org.ovirt.engine.core.utils.timer.SchedulerUtilQuartzImpl] (DefaultQuartzScheduler_Worker-45) Failed to invoke scheduled method _updatingTimer_Elapsed: java.lang.reflect.InvocationTargetException > at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source) [:1.7.0_55] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.lang.NoSuchMethodError: org.ovirt.vdsm.jsonrpc.client.utils.retry.RetryPolicy.<init>(IIILjava/lang/Class;)V > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.getJsonClient(JsonRpcUtils.java:48) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.createClient(JsonRpcUtils.java:38) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.createStompClient(JsonRpcUtils.java:22) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.TransportFactory.createIrsServer(TransportFactory.java:21) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData.getIrsProxy(IrsProxyData.java:579) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand.getIrsProxy(IrsBrokerCommand.java:146) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand.executeVDSCommand(IrsBrokerCommand.java:154) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:407) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData.proceedStoragePoolStats(IrsProxyData.java:293) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData._updatingTimer_Elapsed(IrsProxyData.java:187) [vdsbroker.jar:] > ... 6 more > >2014-08-11 16:04:59,463 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-19) [ffcc4f7] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-08-11 16:04:59,567 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-19) [ffcc4f7] hostFromVds::selectedVds - rack, spmStatus SPM, storage pool Default >2014-08-11 16:04:59,572 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-19) [ffcc4f7] Initialize Irs proxy from vds: 10.34.62.208 >2014-08-11 16:04:59,591 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-19) [ffcc4f7] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208). >2014-08-11 16:04:59,595 ERROR [org.ovirt.engine.core.utils.timer.SchedulerUtilQuartzImpl] (DefaultQuartzScheduler_Worker-19) Failed to invoke scheduled method _updatingTimer_Elapsed: java.lang.reflect.InvocationTargetException > at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source) [:1.7.0_55] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.lang.NoSuchMethodError: org.ovirt.vdsm.jsonrpc.client.utils.retry.RetryPolicy.<init>(IIILjava/lang/Class;)V > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.getJsonClient(JsonRpcUtils.java:48) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.createClient(JsonRpcUtils.java:38) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.createStompClient(JsonRpcUtils.java:22) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.TransportFactory.createIrsServer(TransportFactory.java:21) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData.getIrsProxy(IrsProxyData.java:579) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand.getIrsProxy(IrsBrokerCommand.java:146) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand.executeVDSCommand(IrsBrokerCommand.java:154) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:407) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData.proceedStoragePoolStats(IrsProxyData.java:293) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData._updatingTimer_Elapsed(IrsProxyData.java:187) [vdsbroker.jar:] > ... 6 more > >2014-08-11 16:05:09,705 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-35) [1d849b94] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-08-11 16:05:09,789 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-35) [1d849b94] hostFromVds::selectedVds - rack, spmStatus SPM, storage pool Default >2014-08-11 16:05:09,792 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-35) [1d849b94] Initialize Irs proxy from vds: 10.34.62.208 >2014-08-11 16:05:09,796 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-35) [1d849b94] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208). >2014-08-11 16:05:09,797 ERROR [org.ovirt.engine.core.utils.timer.SchedulerUtilQuartzImpl] (DefaultQuartzScheduler_Worker-35) Failed to invoke scheduled method _updatingTimer_Elapsed: java.lang.reflect.InvocationTargetException > at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source) [:1.7.0_55] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.lang.NoSuchMethodError: org.ovirt.vdsm.jsonrpc.client.utils.retry.RetryPolicy.<init>(IIILjava/lang/Class;)V > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.getJsonClient(JsonRpcUtils.java:48) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.createClient(JsonRpcUtils.java:38) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.createStompClient(JsonRpcUtils.java:22) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.TransportFactory.createIrsServer(TransportFactory.java:21) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData.getIrsProxy(IrsProxyData.java:579) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand.getIrsProxy(IrsBrokerCommand.java:146) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand.executeVDSCommand(IrsBrokerCommand.java:154) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:407) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData.proceedStoragePoolStats(IrsProxyData.java:293) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData._updatingTimer_Elapsed(IrsProxyData.java:187) [vdsbroker.jar:] > ... 6 more > >2014-08-11 16:05:19,884 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-76) [59838f6f] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-08-11 16:05:19,895 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-76) [59838f6f] Correlation ID: 59838f6f, Call Stack: null, Custom Event ID: -1, Message: Data Center is being initialized, please wait for initialization to complete. >2014-08-11 16:05:19,972 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-76) [59838f6f] hostFromVds::selectedVds - rack, spmStatus SPM, storage pool Default >2014-08-11 16:05:19,977 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-76) [59838f6f] Initialize Irs proxy from vds: 10.34.62.208 >2014-08-11 16:05:19,984 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-76) [59838f6f] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208). >2014-08-11 16:05:19,985 ERROR [org.ovirt.engine.core.utils.timer.SchedulerUtilQuartzImpl] (DefaultQuartzScheduler_Worker-76) Failed to invoke scheduled method _updatingTimer_Elapsed: java.lang.reflect.InvocationTargetException > at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source) [:1.7.0_55] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.lang.NoSuchMethodError: org.ovirt.vdsm.jsonrpc.client.utils.retry.RetryPolicy.<init>(IIILjava/lang/Class;)V > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.getJsonClient(JsonRpcUtils.java:48) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.createClient(JsonRpcUtils.java:38) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.createStompClient(JsonRpcUtils.java:22) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.TransportFactory.createIrsServer(TransportFactory.java:21) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData.getIrsProxy(IrsProxyData.java:579) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand.getIrsProxy(IrsBrokerCommand.java:146) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand.executeVDSCommand(IrsBrokerCommand.java:154) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:407) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData.proceedStoragePoolStats(IrsProxyData.java:293) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData._updatingTimer_Elapsed(IrsProxyData.java:187) [vdsbroker.jar:] > ... 6 more > >2014-08-11 16:05:30,043 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-12) [d351fc8] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-08-11 16:05:30,109 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-12) [d351fc8] hostFromVds::selectedVds - rack, spmStatus SPM, storage pool Default >2014-08-11 16:05:30,112 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-12) [d351fc8] Initialize Irs proxy from vds: 10.34.62.208 >2014-08-11 16:05:30,118 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-12) [d351fc8] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208). >2014-08-11 16:05:30,119 ERROR [org.ovirt.engine.core.utils.timer.SchedulerUtilQuartzImpl] (DefaultQuartzScheduler_Worker-12) Failed to invoke scheduled method _updatingTimer_Elapsed: java.lang.reflect.InvocationTargetException > at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source) [:1.7.0_55] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.lang.NoSuchMethodError: org.ovirt.vdsm.jsonrpc.client.utils.retry.RetryPolicy.<init>(IIILjava/lang/Class;)V > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.getJsonClient(JsonRpcUtils.java:48) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.createClient(JsonRpcUtils.java:38) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.createStompClient(JsonRpcUtils.java:22) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.TransportFactory.createIrsServer(TransportFactory.java:21) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData.getIrsProxy(IrsProxyData.java:579) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand.getIrsProxy(IrsBrokerCommand.java:146) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand.executeVDSCommand(IrsBrokerCommand.java:154) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:407) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData.proceedStoragePoolStats(IrsProxyData.java:293) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData._updatingTimer_Elapsed(IrsProxyData.java:187) [vdsbroker.jar:] > ... 6 more > >2014-08-11 16:05:40,171 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-100) [10593fed] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-08-11 16:05:40,236 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-100) [10593fed] hostFromVds::selectedVds - rack, spmStatus SPM, storage pool Default >2014-08-11 16:05:40,240 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-100) [10593fed] Initialize Irs proxy from vds: 10.34.62.208 >2014-08-11 16:05:40,247 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-100) [10593fed] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208). >2014-08-11 16:05:40,249 ERROR [org.ovirt.engine.core.utils.timer.SchedulerUtilQuartzImpl] (DefaultQuartzScheduler_Worker-100) Failed to invoke scheduled method _updatingTimer_Elapsed: java.lang.reflect.InvocationTargetException > at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source) [:1.7.0_55] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.lang.NoSuchMethodError: org.ovirt.vdsm.jsonrpc.client.utils.retry.RetryPolicy.<init>(IIILjava/lang/Class;)V > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.getJsonClient(JsonRpcUtils.java:48) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.createClient(JsonRpcUtils.java:38) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.createStompClient(JsonRpcUtils.java:22) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.TransportFactory.createIrsServer(TransportFactory.java:21) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData.getIrsProxy(IrsProxyData.java:579) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand.getIrsProxy(IrsBrokerCommand.java:146) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand.executeVDSCommand(IrsBrokerCommand.java:154) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:407) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData.proceedStoragePoolStats(IrsProxyData.java:293) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData._updatingTimer_Elapsed(IrsProxyData.java:187) [vdsbroker.jar:] > ... 6 more > >2014-08-11 16:05:50,300 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-29) [1ebede43] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-08-11 16:05:50,399 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-29) [1ebede43] hostFromVds::selectedVds - rack, spmStatus SPM, storage pool Default >2014-08-11 16:05:50,403 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-29) [1ebede43] Initialize Irs proxy from vds: 10.34.62.208 >2014-08-11 16:05:50,409 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-29) [1ebede43] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208). >2014-08-11 16:05:50,410 ERROR [org.ovirt.engine.core.utils.timer.SchedulerUtilQuartzImpl] (DefaultQuartzScheduler_Worker-29) Failed to invoke scheduled method _updatingTimer_Elapsed: java.lang.reflect.InvocationTargetException > at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source) [:1.7.0_55] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.lang.NoSuchMethodError: org.ovirt.vdsm.jsonrpc.client.utils.retry.RetryPolicy.<init>(IIILjava/lang/Class;)V > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.getJsonClient(JsonRpcUtils.java:48) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.createClient(JsonRpcUtils.java:38) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.createStompClient(JsonRpcUtils.java:22) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.TransportFactory.createIrsServer(TransportFactory.java:21) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData.getIrsProxy(IrsProxyData.java:579) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand.getIrsProxy(IrsBrokerCommand.java:146) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand.executeVDSCommand(IrsBrokerCommand.java:154) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:407) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData.proceedStoragePoolStats(IrsProxyData.java:293) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData._updatingTimer_Elapsed(IrsProxyData.java:187) [vdsbroker.jar:] > ... 6 more > >2014-08-11 16:06:00,467 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-84) [5b2fa51b] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-08-11 16:06:00,534 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-84) [5b2fa51b] hostFromVds::selectedVds - rack, spmStatus SPM, storage pool Default >2014-08-11 16:06:00,538 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-84) [5b2fa51b] Initialize Irs proxy from vds: 10.34.62.208 >2014-08-11 16:06:00,544 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-84) [5b2fa51b] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208). >2014-08-11 16:06:00,545 ERROR [org.ovirt.engine.core.utils.timer.SchedulerUtilQuartzImpl] (DefaultQuartzScheduler_Worker-84) Failed to invoke scheduled method _updatingTimer_Elapsed: java.lang.reflect.InvocationTargetException > at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source) [:1.7.0_55] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.lang.NoSuchMethodError: org.ovirt.vdsm.jsonrpc.client.utils.retry.RetryPolicy.<init>(IIILjava/lang/Class;)V > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.getJsonClient(JsonRpcUtils.java:48) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.createClient(JsonRpcUtils.java:38) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.createStompClient(JsonRpcUtils.java:22) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.TransportFactory.createIrsServer(TransportFactory.java:21) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData.getIrsProxy(IrsProxyData.java:579) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand.getIrsProxy(IrsBrokerCommand.java:146) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand.executeVDSCommand(IrsBrokerCommand.java:154) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:407) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData.proceedStoragePoolStats(IrsProxyData.java:293) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData._updatingTimer_Elapsed(IrsProxyData.java:187) [vdsbroker.jar:] > ... 6 more > >2014-08-11 16:06:10,604 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-64) [3dd51269] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-08-11 16:06:10,669 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-64) [3dd51269] hostFromVds::selectedVds - rack, spmStatus SPM, storage pool Default >2014-08-11 16:06:10,672 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-64) [3dd51269] Initialize Irs proxy from vds: 10.34.62.208 >2014-08-11 16:06:10,679 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-64) [3dd51269] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208). >2014-08-11 16:06:10,680 ERROR [org.ovirt.engine.core.utils.timer.SchedulerUtilQuartzImpl] (DefaultQuartzScheduler_Worker-64) Failed to invoke scheduled method _updatingTimer_Elapsed: java.lang.reflect.InvocationTargetException > at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source) [:1.7.0_55] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.lang.NoSuchMethodError: org.ovirt.vdsm.jsonrpc.client.utils.retry.RetryPolicy.<init>(IIILjava/lang/Class;)V > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.getJsonClient(JsonRpcUtils.java:48) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.createClient(JsonRpcUtils.java:38) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.createStompClient(JsonRpcUtils.java:22) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.TransportFactory.createIrsServer(TransportFactory.java:21) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData.getIrsProxy(IrsProxyData.java:579) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand.getIrsProxy(IrsBrokerCommand.java:146) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand.executeVDSCommand(IrsBrokerCommand.java:154) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:407) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData.proceedStoragePoolStats(IrsProxyData.java:293) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData._updatingTimer_Elapsed(IrsProxyData.java:187) [vdsbroker.jar:] > ... 6 more > >2014-08-11 16:06:20,736 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-6) [30a5360c] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-08-11 16:06:20,747 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-6) [30a5360c] Correlation ID: 30a5360c, Call Stack: null, Custom Event ID: -1, Message: Data Center is being initialized, please wait for initialization to complete. >2014-08-11 16:06:20,805 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-6) [30a5360c] hostFromVds::selectedVds - rack, spmStatus SPM, storage pool Default >2014-08-11 16:06:20,809 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-6) [30a5360c] Initialize Irs proxy from vds: 10.34.62.208 >2014-08-11 16:06:20,815 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-6) [30a5360c] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208). >2014-08-11 16:06:20,816 ERROR [org.ovirt.engine.core.utils.timer.SchedulerUtilQuartzImpl] (DefaultQuartzScheduler_Worker-6) Failed to invoke scheduled method _updatingTimer_Elapsed: java.lang.reflect.InvocationTargetException > at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source) [:1.7.0_55] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.lang.NoSuchMethodError: org.ovirt.vdsm.jsonrpc.client.utils.retry.RetryPolicy.<init>(IIILjava/lang/Class;)V > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.getJsonClient(JsonRpcUtils.java:48) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.createClient(JsonRpcUtils.java:38) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.createStompClient(JsonRpcUtils.java:22) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.TransportFactory.createIrsServer(TransportFactory.java:21) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData.getIrsProxy(IrsProxyData.java:579) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand.getIrsProxy(IrsBrokerCommand.java:146) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand.executeVDSCommand(IrsBrokerCommand.java:154) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:407) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData.proceedStoragePoolStats(IrsProxyData.java:293) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData._updatingTimer_Elapsed(IrsProxyData.java:187) [vdsbroker.jar:] > ... 6 more > >2014-08-11 16:06:30,873 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-81) [19324c69] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-08-11 16:06:30,939 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-81) [19324c69] hostFromVds::selectedVds - rack, spmStatus SPM, storage pool Default >2014-08-11 16:06:30,942 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-81) [19324c69] Initialize Irs proxy from vds: 10.34.62.208 >2014-08-11 16:06:30,965 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-81) [19324c69] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208). >2014-08-11 16:06:30,966 ERROR [org.ovirt.engine.core.utils.timer.SchedulerUtilQuartzImpl] (DefaultQuartzScheduler_Worker-81) Failed to invoke scheduled method _updatingTimer_Elapsed: java.lang.reflect.InvocationTargetException > at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source) [:1.7.0_55] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.lang.NoSuchMethodError: org.ovirt.vdsm.jsonrpc.client.utils.retry.RetryPolicy.<init>(IIILjava/lang/Class;)V > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.getJsonClient(JsonRpcUtils.java:48) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.createClient(JsonRpcUtils.java:38) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.createStompClient(JsonRpcUtils.java:22) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.TransportFactory.createIrsServer(TransportFactory.java:21) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData.getIrsProxy(IrsProxyData.java:579) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand.getIrsProxy(IrsBrokerCommand.java:146) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand.executeVDSCommand(IrsBrokerCommand.java:154) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:407) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData.proceedStoragePoolStats(IrsProxyData.java:293) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData._updatingTimer_Elapsed(IrsProxyData.java:187) [vdsbroker.jar:] > ... 6 more > >2014-08-11 16:06:41,019 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-37) [5147fe99] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-08-11 16:06:41,089 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-37) [5147fe99] hostFromVds::selectedVds - rack, spmStatus SPM, storage pool Default >2014-08-11 16:06:41,093 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-37) [5147fe99] Initialize Irs proxy from vds: 10.34.62.208 >2014-08-11 16:06:41,100 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-37) [5147fe99] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208). >2014-08-11 16:06:41,102 ERROR [org.ovirt.engine.core.utils.timer.SchedulerUtilQuartzImpl] (DefaultQuartzScheduler_Worker-37) Failed to invoke scheduled method _updatingTimer_Elapsed: java.lang.reflect.InvocationTargetException > at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source) [:1.7.0_55] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.lang.NoSuchMethodError: org.ovirt.vdsm.jsonrpc.client.utils.retry.RetryPolicy.<init>(IIILjava/lang/Class;)V > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.getJsonClient(JsonRpcUtils.java:48) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.createClient(JsonRpcUtils.java:38) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.createStompClient(JsonRpcUtils.java:22) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.TransportFactory.createIrsServer(TransportFactory.java:21) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData.getIrsProxy(IrsProxyData.java:579) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand.getIrsProxy(IrsBrokerCommand.java:146) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand.executeVDSCommand(IrsBrokerCommand.java:154) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:407) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData.proceedStoragePoolStats(IrsProxyData.java:293) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData._updatingTimer_Elapsed(IrsProxyData.java:187) [vdsbroker.jar:] > ... 6 more > >2014-08-11 16:06:51,171 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-73) [7ec672a] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-08-11 16:06:51,242 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-73) [7ec672a] hostFromVds::selectedVds - rack, spmStatus SPM, storage pool Default >2014-08-11 16:06:51,247 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-73) [7ec672a] Initialize Irs proxy from vds: 10.34.62.208 >2014-08-11 16:06:51,257 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-73) [7ec672a] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208). >2014-08-11 16:06:51,258 ERROR [org.ovirt.engine.core.utils.timer.SchedulerUtilQuartzImpl] (DefaultQuartzScheduler_Worker-73) Failed to invoke scheduled method _updatingTimer_Elapsed: java.lang.reflect.InvocationTargetException > at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source) [:1.7.0_55] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.lang.NoSuchMethodError: org.ovirt.vdsm.jsonrpc.client.utils.retry.RetryPolicy.<init>(IIILjava/lang/Class;)V > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.getJsonClient(JsonRpcUtils.java:48) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.createClient(JsonRpcUtils.java:38) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.createStompClient(JsonRpcUtils.java:22) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.TransportFactory.createIrsServer(TransportFactory.java:21) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData.getIrsProxy(IrsProxyData.java:579) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand.getIrsProxy(IrsBrokerCommand.java:146) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand.executeVDSCommand(IrsBrokerCommand.java:154) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:407) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData.proceedStoragePoolStats(IrsProxyData.java:293) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData._updatingTimer_Elapsed(IrsProxyData.java:187) [vdsbroker.jar:] > ... 6 more > >2014-08-11 16:07:01,318 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-77) [2daf0726] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-08-11 16:07:01,385 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-77) [2daf0726] hostFromVds::selectedVds - rack, spmStatus SPM, storage pool Default >2014-08-11 16:07:01,390 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-77) [2daf0726] Initialize Irs proxy from vds: 10.34.62.208 >2014-08-11 16:07:01,397 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-77) [2daf0726] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208). >2014-08-11 16:07:01,399 ERROR [org.ovirt.engine.core.utils.timer.SchedulerUtilQuartzImpl] (DefaultQuartzScheduler_Worker-77) Failed to invoke scheduled method _updatingTimer_Elapsed: java.lang.reflect.InvocationTargetException > at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source) [:1.7.0_55] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.lang.NoSuchMethodError: org.ovirt.vdsm.jsonrpc.client.utils.retry.RetryPolicy.<init>(IIILjava/lang/Class;)V > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.getJsonClient(JsonRpcUtils.java:48) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.createClient(JsonRpcUtils.java:38) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.createStompClient(JsonRpcUtils.java:22) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.TransportFactory.createIrsServer(TransportFactory.java:21) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData.getIrsProxy(IrsProxyData.java:579) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand.getIrsProxy(IrsBrokerCommand.java:146) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand.executeVDSCommand(IrsBrokerCommand.java:154) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:407) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData.proceedStoragePoolStats(IrsProxyData.java:293) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData._updatingTimer_Elapsed(IrsProxyData.java:187) [vdsbroker.jar:] > ... 6 more > >2014-08-11 16:07:11,586 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-57) [28141123] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-08-11 16:07:11,671 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-57) [28141123] hostFromVds::selectedVds - rack, spmStatus SPM, storage pool Default >2014-08-11 16:07:11,677 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-57) [28141123] Initialize Irs proxy from vds: 10.34.62.208 >2014-08-11 16:07:11,684 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-57) [28141123] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208). >2014-08-11 16:07:11,686 ERROR [org.ovirt.engine.core.utils.timer.SchedulerUtilQuartzImpl] (DefaultQuartzScheduler_Worker-57) Failed to invoke scheduled method _updatingTimer_Elapsed: java.lang.reflect.InvocationTargetException > at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source) [:1.7.0_55] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.lang.NoSuchMethodError: org.ovirt.vdsm.jsonrpc.client.utils.retry.RetryPolicy.<init>(IIILjava/lang/Class;)V > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.getJsonClient(JsonRpcUtils.java:48) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.createClient(JsonRpcUtils.java:38) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.createStompClient(JsonRpcUtils.java:22) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.TransportFactory.createIrsServer(TransportFactory.java:21) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData.getIrsProxy(IrsProxyData.java:579) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand.getIrsProxy(IrsBrokerCommand.java:146) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand.executeVDSCommand(IrsBrokerCommand.java:154) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:407) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData.proceedStoragePoolStats(IrsProxyData.java:293) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData._updatingTimer_Elapsed(IrsProxyData.java:187) [vdsbroker.jar:] > ... 6 more > >2014-08-11 16:07:21,745 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-75) [55e964ee] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-08-11 16:07:21,759 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-75) [55e964ee] Correlation ID: 55e964ee, Call Stack: null, Custom Event ID: -1, Message: Data Center is being initialized, please wait for initialization to complete. >2014-08-11 16:07:21,778 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-75) [55e964ee] hostFromVds::selectedVds - rack, spmStatus SPM, storage pool Default >2014-08-11 16:07:21,782 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-75) [55e964ee] Initialize Irs proxy from vds: 10.34.62.208 >2014-08-11 16:07:21,789 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-75) [55e964ee] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208). >2014-08-11 16:07:21,791 ERROR [org.ovirt.engine.core.utils.timer.SchedulerUtilQuartzImpl] (DefaultQuartzScheduler_Worker-75) Failed to invoke scheduled method _updatingTimer_Elapsed: java.lang.reflect.InvocationTargetException > at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source) [:1.7.0_55] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.lang.NoSuchMethodError: org.ovirt.vdsm.jsonrpc.client.utils.retry.RetryPolicy.<init>(IIILjava/lang/Class;)V > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.getJsonClient(JsonRpcUtils.java:48) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.createClient(JsonRpcUtils.java:38) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.createStompClient(JsonRpcUtils.java:22) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.TransportFactory.createIrsServer(TransportFactory.java:21) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData.getIrsProxy(IrsProxyData.java:579) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand.getIrsProxy(IrsBrokerCommand.java:146) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand.executeVDSCommand(IrsBrokerCommand.java:154) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:407) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData.proceedStoragePoolStats(IrsProxyData.java:293) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData._updatingTimer_Elapsed(IrsProxyData.java:187) [vdsbroker.jar:] > ... 6 more > >2014-08-11 16:07:31,851 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-49) [64682f24] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-08-11 16:07:31,916 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-49) [64682f24] hostFromVds::selectedVds - rack, spmStatus SPM, storage pool Default >2014-08-11 16:07:31,919 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-49) [64682f24] Initialize Irs proxy from vds: 10.34.62.208 >2014-08-11 16:07:31,925 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-49) [64682f24] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208). >2014-08-11 16:07:31,927 ERROR [org.ovirt.engine.core.utils.timer.SchedulerUtilQuartzImpl] (DefaultQuartzScheduler_Worker-49) Failed to invoke scheduled method _updatingTimer_Elapsed: java.lang.reflect.InvocationTargetException > at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source) [:1.7.0_55] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.lang.NoSuchMethodError: org.ovirt.vdsm.jsonrpc.client.utils.retry.RetryPolicy.<init>(IIILjava/lang/Class;)V > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.getJsonClient(JsonRpcUtils.java:48) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.createClient(JsonRpcUtils.java:38) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.createStompClient(JsonRpcUtils.java:22) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.TransportFactory.createIrsServer(TransportFactory.java:21) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData.getIrsProxy(IrsProxyData.java:579) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand.getIrsProxy(IrsBrokerCommand.java:146) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand.executeVDSCommand(IrsBrokerCommand.java:154) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:407) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData.proceedStoragePoolStats(IrsProxyData.java:293) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData._updatingTimer_Elapsed(IrsProxyData.java:187) [vdsbroker.jar:] > ... 6 more > >2014-08-11 16:07:41,982 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-92) [2707196a] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-08-11 16:07:42,049 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-92) [2707196a] hostFromVds::selectedVds - rack, spmStatus SPM, storage pool Default >2014-08-11 16:07:42,052 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-92) [2707196a] Initialize Irs proxy from vds: 10.34.62.208 >2014-08-11 16:07:42,058 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-92) [2707196a] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208). >2014-08-11 16:07:42,060 ERROR [org.ovirt.engine.core.utils.timer.SchedulerUtilQuartzImpl] (DefaultQuartzScheduler_Worker-92) Failed to invoke scheduled method _updatingTimer_Elapsed: java.lang.reflect.InvocationTargetException > at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source) [:1.7.0_55] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.lang.NoSuchMethodError: org.ovirt.vdsm.jsonrpc.client.utils.retry.RetryPolicy.<init>(IIILjava/lang/Class;)V > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.getJsonClient(JsonRpcUtils.java:48) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.createClient(JsonRpcUtils.java:38) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.createStompClient(JsonRpcUtils.java:22) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.TransportFactory.createIrsServer(TransportFactory.java:21) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData.getIrsProxy(IrsProxyData.java:579) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand.getIrsProxy(IrsBrokerCommand.java:146) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand.executeVDSCommand(IrsBrokerCommand.java:154) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:407) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData.proceedStoragePoolStats(IrsProxyData.java:293) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData._updatingTimer_Elapsed(IrsProxyData.java:187) [vdsbroker.jar:] > ... 6 more > >2014-08-11 16:07:52,115 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-76) [43def580] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-08-11 16:07:52,181 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-76) [43def580] hostFromVds::selectedVds - rack, spmStatus SPM, storage pool Default >2014-08-11 16:07:52,185 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-76) [43def580] Initialize Irs proxy from vds: 10.34.62.208 >2014-08-11 16:07:52,192 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-76) [43def580] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208). >2014-08-11 16:07:52,193 ERROR [org.ovirt.engine.core.utils.timer.SchedulerUtilQuartzImpl] (DefaultQuartzScheduler_Worker-76) Failed to invoke scheduled method _updatingTimer_Elapsed: java.lang.reflect.InvocationTargetException > at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source) [:1.7.0_55] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.lang.NoSuchMethodError: org.ovirt.vdsm.jsonrpc.client.utils.retry.RetryPolicy.<init>(IIILjava/lang/Class;)V > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.getJsonClient(JsonRpcUtils.java:48) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.createClient(JsonRpcUtils.java:38) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.createStompClient(JsonRpcUtils.java:22) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.TransportFactory.createIrsServer(TransportFactory.java:21) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData.getIrsProxy(IrsProxyData.java:579) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand.getIrsProxy(IrsBrokerCommand.java:146) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand.executeVDSCommand(IrsBrokerCommand.java:154) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:407) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData.proceedStoragePoolStats(IrsProxyData.java:293) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData._updatingTimer_Elapsed(IrsProxyData.java:187) [vdsbroker.jar:] > ... 6 more > >2014-08-11 16:08:02,252 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-24) [10064a3d] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-08-11 16:08:02,317 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-24) [10064a3d] hostFromVds::selectedVds - rack, spmStatus SPM, storage pool Default >2014-08-11 16:08:02,320 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-24) [10064a3d] Initialize Irs proxy from vds: 10.34.62.208 >2014-08-11 16:08:02,327 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-24) [10064a3d] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208). >2014-08-11 16:08:02,328 ERROR [org.ovirt.engine.core.utils.timer.SchedulerUtilQuartzImpl] (DefaultQuartzScheduler_Worker-24) Failed to invoke scheduled method _updatingTimer_Elapsed: java.lang.reflect.InvocationTargetException > at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source) [:1.7.0_55] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.lang.NoSuchMethodError: org.ovirt.vdsm.jsonrpc.client.utils.retry.RetryPolicy.<init>(IIILjava/lang/Class;)V > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.getJsonClient(JsonRpcUtils.java:48) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.createClient(JsonRpcUtils.java:38) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.createStompClient(JsonRpcUtils.java:22) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.TransportFactory.createIrsServer(TransportFactory.java:21) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData.getIrsProxy(IrsProxyData.java:579) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand.getIrsProxy(IrsBrokerCommand.java:146) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand.executeVDSCommand(IrsBrokerCommand.java:154) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:407) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData.proceedStoragePoolStats(IrsProxyData.java:293) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData._updatingTimer_Elapsed(IrsProxyData.java:187) [vdsbroker.jar:] > ... 6 more > >2014-08-11 16:08:12,415 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-13) [5a1e10d6] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-08-11 16:08:12,484 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-13) [5a1e10d6] hostFromVds::selectedVds - rack, spmStatus SPM, storage pool Default >2014-08-11 16:08:12,488 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-13) [5a1e10d6] Initialize Irs proxy from vds: 10.34.62.208 >2014-08-11 16:08:12,495 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-13) [5a1e10d6] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208). >2014-08-11 16:08:12,496 ERROR [org.ovirt.engine.core.utils.timer.SchedulerUtilQuartzImpl] (DefaultQuartzScheduler_Worker-13) Failed to invoke scheduled method _updatingTimer_Elapsed: java.lang.reflect.InvocationTargetException > at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source) [:1.7.0_55] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.lang.NoSuchMethodError: org.ovirt.vdsm.jsonrpc.client.utils.retry.RetryPolicy.<init>(IIILjava/lang/Class;)V > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.getJsonClient(JsonRpcUtils.java:48) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.createClient(JsonRpcUtils.java:38) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.createStompClient(JsonRpcUtils.java:22) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.TransportFactory.createIrsServer(TransportFactory.java:21) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData.getIrsProxy(IrsProxyData.java:579) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand.getIrsProxy(IrsBrokerCommand.java:146) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand.executeVDSCommand(IrsBrokerCommand.java:154) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:407) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData.proceedStoragePoolStats(IrsProxyData.java:293) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData._updatingTimer_Elapsed(IrsProxyData.java:187) [vdsbroker.jar:] > ... 6 more > >2014-08-11 16:08:22,565 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-78) [2fc9b3f7] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-08-11 16:08:22,579 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-78) [2fc9b3f7] Correlation ID: 2fc9b3f7, Call Stack: null, Custom Event ID: -1, Message: Data Center is being initialized, please wait for initialization to complete. >2014-08-11 16:08:22,702 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-78) [2fc9b3f7] hostFromVds::selectedVds - rack, spmStatus SPM, storage pool Default >2014-08-11 16:08:22,706 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-78) [2fc9b3f7] Initialize Irs proxy from vds: 10.34.62.208 >2014-08-11 16:08:22,713 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-78) [2fc9b3f7] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208). >2014-08-11 16:08:22,716 ERROR [org.ovirt.engine.core.utils.timer.SchedulerUtilQuartzImpl] (DefaultQuartzScheduler_Worker-78) Failed to invoke scheduled method _updatingTimer_Elapsed: java.lang.reflect.InvocationTargetException > at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source) [:1.7.0_55] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.lang.NoSuchMethodError: org.ovirt.vdsm.jsonrpc.client.utils.retry.RetryPolicy.<init>(IIILjava/lang/Class;)V > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.getJsonClient(JsonRpcUtils.java:48) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.createClient(JsonRpcUtils.java:38) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.createStompClient(JsonRpcUtils.java:22) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.TransportFactory.createIrsServer(TransportFactory.java:21) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData.getIrsProxy(IrsProxyData.java:579) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand.getIrsProxy(IrsBrokerCommand.java:146) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand.executeVDSCommand(IrsBrokerCommand.java:154) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:407) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData.proceedStoragePoolStats(IrsProxyData.java:293) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData._updatingTimer_Elapsed(IrsProxyData.java:187) [vdsbroker.jar:] > ... 6 more > >2014-08-11 16:08:32,778 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-7) [306c8cb8] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-08-11 16:08:32,805 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-7) [306c8cb8] hostFromVds::selectedVds - rack, spmStatus SPM, storage pool Default >2014-08-11 16:08:32,809 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-7) [306c8cb8] Initialize Irs proxy from vds: 10.34.62.208 >2014-08-11 16:08:32,816 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-7) [306c8cb8] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208). >2014-08-11 16:08:32,818 ERROR [org.ovirt.engine.core.utils.timer.SchedulerUtilQuartzImpl] (DefaultQuartzScheduler_Worker-7) Failed to invoke scheduled method _updatingTimer_Elapsed: java.lang.reflect.InvocationTargetException > at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source) [:1.7.0_55] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.lang.NoSuchMethodError: org.ovirt.vdsm.jsonrpc.client.utils.retry.RetryPolicy.<init>(IIILjava/lang/Class;)V > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.getJsonClient(JsonRpcUtils.java:48) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.createClient(JsonRpcUtils.java:38) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.createStompClient(JsonRpcUtils.java:22) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.TransportFactory.createIrsServer(TransportFactory.java:21) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData.getIrsProxy(IrsProxyData.java:579) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand.getIrsProxy(IrsBrokerCommand.java:146) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand.executeVDSCommand(IrsBrokerCommand.java:154) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:407) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData.proceedStoragePoolStats(IrsProxyData.java:293) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData._updatingTimer_Elapsed(IrsProxyData.java:187) [vdsbroker.jar:] > ... 6 more > >2014-08-11 16:08:42,871 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-83) [507a5afd] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-08-11 16:08:42,937 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-83) [507a5afd] hostFromVds::selectedVds - rack, spmStatus SPM, storage pool Default >2014-08-11 16:08:42,945 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-83) [507a5afd] Initialize Irs proxy from vds: 10.34.62.208 >2014-08-11 16:08:42,954 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-83) [507a5afd] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208). >2014-08-11 16:08:42,955 ERROR [org.ovirt.engine.core.utils.timer.SchedulerUtilQuartzImpl] (DefaultQuartzScheduler_Worker-83) Failed to invoke scheduled method _updatingTimer_Elapsed: java.lang.reflect.InvocationTargetException > at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source) [:1.7.0_55] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.lang.NoSuchMethodError: org.ovirt.vdsm.jsonrpc.client.utils.retry.RetryPolicy.<init>(IIILjava/lang/Class;)V > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.getJsonClient(JsonRpcUtils.java:48) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.createClient(JsonRpcUtils.java:38) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.createStompClient(JsonRpcUtils.java:22) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.TransportFactory.createIrsServer(TransportFactory.java:21) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData.getIrsProxy(IrsProxyData.java:579) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand.getIrsProxy(IrsBrokerCommand.java:146) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand.executeVDSCommand(IrsBrokerCommand.java:154) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:407) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData.proceedStoragePoolStats(IrsProxyData.java:293) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData._updatingTimer_Elapsed(IrsProxyData.java:187) [vdsbroker.jar:] > ... 6 more > >2014-08-11 16:08:53,016 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-89) [7352f1e1] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-08-11 16:08:53,081 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-89) [7352f1e1] hostFromVds::selectedVds - rack, spmStatus SPM, storage pool Default >2014-08-11 16:08:53,087 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-89) [7352f1e1] Initialize Irs proxy from vds: 10.34.62.208 >2014-08-11 16:08:53,097 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-89) [7352f1e1] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208). >2014-08-11 16:08:53,099 ERROR [org.ovirt.engine.core.utils.timer.SchedulerUtilQuartzImpl] (DefaultQuartzScheduler_Worker-89) Failed to invoke scheduled method _updatingTimer_Elapsed: java.lang.reflect.InvocationTargetException > at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source) [:1.7.0_55] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.lang.NoSuchMethodError: org.ovirt.vdsm.jsonrpc.client.utils.retry.RetryPolicy.<init>(IIILjava/lang/Class;)V > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.getJsonClient(JsonRpcUtils.java:48) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.createClient(JsonRpcUtils.java:38) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.createStompClient(JsonRpcUtils.java:22) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.TransportFactory.createIrsServer(TransportFactory.java:21) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData.getIrsProxy(IrsProxyData.java:579) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand.getIrsProxy(IrsBrokerCommand.java:146) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand.executeVDSCommand(IrsBrokerCommand.java:154) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:407) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData.proceedStoragePoolStats(IrsProxyData.java:293) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData._updatingTimer_Elapsed(IrsProxyData.java:187) [vdsbroker.jar:] > ... 6 more > >2014-08-11 16:09:02,450 INFO [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (ajp--127.0.0.1-8702-2) [fde28b8] Lock Acquired to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE >, sharedLocks= ] >2014-08-11 16:09:02,468 WARN [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (ajp--127.0.0.1-8702-2) [fde28b8] CanDoAction of action ActivateStorageDomain failed. Reasons:VAR__TYPE__STORAGE__DOMAIN,VAR__ACTION__ACTIVATE,ACTION_TYPE_FAILED_MASTER_STORAGE_DOMAIN_NOT_ACTIVE >2014-08-11 16:09:02,471 INFO [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (ajp--127.0.0.1-8702-2) [fde28b8] Lock freed to object EngineLock [exclusiveLocks= key: 44559ec0-6bef-491c-bbd6-e67278f37987 value: STORAGE >, sharedLocks= ] >2014-08-11 16:09:03,155 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-31) [4c7ecba8] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-08-11 16:09:03,223 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-31) [4c7ecba8] hostFromVds::selectedVds - rack, spmStatus SPM, storage pool Default >2014-08-11 16:09:03,227 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-31) [4c7ecba8] Initialize Irs proxy from vds: 10.34.62.208 >2014-08-11 16:09:03,234 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-31) [4c7ecba8] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208). >2014-08-11 16:09:03,235 ERROR [org.ovirt.engine.core.utils.timer.SchedulerUtilQuartzImpl] (DefaultQuartzScheduler_Worker-31) Failed to invoke scheduled method _updatingTimer_Elapsed: java.lang.reflect.InvocationTargetException > at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source) [:1.7.0_55] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.lang.NoSuchMethodError: org.ovirt.vdsm.jsonrpc.client.utils.retry.RetryPolicy.<init>(IIILjava/lang/Class;)V > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.getJsonClient(JsonRpcUtils.java:48) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.createClient(JsonRpcUtils.java:38) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.createStompClient(JsonRpcUtils.java:22) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.TransportFactory.createIrsServer(TransportFactory.java:21) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData.getIrsProxy(IrsProxyData.java:579) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand.getIrsProxy(IrsBrokerCommand.java:146) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand.executeVDSCommand(IrsBrokerCommand.java:154) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:407) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData.proceedStoragePoolStats(IrsProxyData.java:293) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData._updatingTimer_Elapsed(IrsProxyData.java:187) [vdsbroker.jar:] > ... 6 more > >2014-08-11 16:09:13,287 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-66) [6e8cea09] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-08-11 16:09:13,354 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-66) [6e8cea09] hostFromVds::selectedVds - rack, spmStatus SPM, storage pool Default >2014-08-11 16:09:13,358 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-66) [6e8cea09] Initialize Irs proxy from vds: 10.34.62.208 >2014-08-11 16:09:13,365 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-66) [6e8cea09] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208). >2014-08-11 16:09:13,366 ERROR [org.ovirt.engine.core.utils.timer.SchedulerUtilQuartzImpl] (DefaultQuartzScheduler_Worker-66) Failed to invoke scheduled method _updatingTimer_Elapsed: java.lang.reflect.InvocationTargetException > at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source) [:1.7.0_55] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.lang.NoSuchMethodError: org.ovirt.vdsm.jsonrpc.client.utils.retry.RetryPolicy.<init>(IIILjava/lang/Class;)V > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.getJsonClient(JsonRpcUtils.java:48) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.createClient(JsonRpcUtils.java:38) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.createStompClient(JsonRpcUtils.java:22) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.TransportFactory.createIrsServer(TransportFactory.java:21) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData.getIrsProxy(IrsProxyData.java:579) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand.getIrsProxy(IrsBrokerCommand.java:146) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand.executeVDSCommand(IrsBrokerCommand.java:154) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:407) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData.proceedStoragePoolStats(IrsProxyData.java:293) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData._updatingTimer_Elapsed(IrsProxyData.java:187) [vdsbroker.jar:] > ... 6 more > >2014-08-11 16:09:14,858 INFO [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (ajp--127.0.0.1-8702-12) [419a23bc] Lock Acquired to object EngineLock [exclusiveLocks= key: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 value: STORAGE >, sharedLocks= ] >2014-08-11 16:09:14,886 INFO [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-5) [419a23bc] Running command: ActivateStorageDomainCommand internal: false. Entities affected : ID: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 Type: Storage >2014-08-11 16:09:14,892 INFO [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-5) [419a23bc] Lock freed to object EngineLock [exclusiveLocks= key: 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752 value: STORAGE >, sharedLocks= ] >2014-08-11 16:09:14,894 INFO [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-5) [419a23bc] ActivateStorage Domain. Before Connect all hosts to pool. Time:8/11/14 4:09 PM >2014-08-11 16:09:14,899 INFO [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-8-thread-48) [7befc6a6] Running command: ConnectStorageToVdsCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2014-08-11 16:09:14,904 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-48) [7befc6a6] START, ConnectStorageServerVDSCommand(HostName = rack, HostId = 17f7fe56-5f32-40ef-b098-611b84484e0b, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: 5083f370-af09-457b-9b73-3440e8b65a07, connection: 10.34.63.202:/mnt/export/nfs/lv3/pbenas/35, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 36910f4d >2014-08-11 16:09:14,917 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-48) [7befc6a6] FINISH, ConnectStorageServerVDSCommand, return: {5083f370-af09-457b-9b73-3440e8b65a07=0}, log id: 36910f4d >2014-08-11 16:09:14,920 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ActivateStorageDomainVDSCommand] (org.ovirt.thread.pool-8-thread-5) [419a23bc] START, ActivateStorageDomainVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000001d9, ignoreFailoverLimit = false, storageDomainId = 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752), log id: 79339613 >2014-08-11 16:09:14,930 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (org.ovirt.thread.pool-8-thread-5) [306141b7] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-08-11 16:09:15,305 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-5) [306141b7] hostFromVds::selectedVds - rack, spmStatus SPM, storage pool Default >2014-08-11 16:09:15,310 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-5) [306141b7] Initialize Irs proxy from vds: 10.34.62.208 >2014-08-11 16:09:15,317 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-5) [306141b7] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208). >2014-08-11 16:09:15,328 INFO [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-5) [306141b7] Command [id=9a19a899-664b-429f-9fb4-1e8cb37e30ab]: Compensating CHANGED_STATUS_ONLY of org.ovirt.engine.core.common.businessentities.StoragePoolIsoMap; snapshot: EntityStatusSnapshot [id=storagePoolId = 00000002-0002-0002-0002-0000000001d9, storageId = 71f82fcd-4fa6-4f56-b228-f4fa4bd4f752, status=Unknown]. >2014-08-11 16:09:15,338 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-5) [306141b7] Correlation ID: 419a23bc, Job ID: a236ea04-ec22-4bb0-ab95-4a33f15c1a7d, Call Stack: null, Custom Event ID: -1, Message: Failed to activate Storage Domain 35 (Data Center Default) by admin >2014-08-11 16:09:23,419 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-18) [3e35b05] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-08-11 16:09:23,429 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-18) [3e35b05] Correlation ID: 3e35b05, Call Stack: null, Custom Event ID: -1, Message: Data Center is being initialized, please wait for initialization to complete. >2014-08-11 16:09:23,487 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-18) [3e35b05] hostFromVds::selectedVds - rack, spmStatus SPM, storage pool Default >2014-08-11 16:09:23,491 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-18) [3e35b05] Initialize Irs proxy from vds: 10.34.62.208 >2014-08-11 16:09:23,498 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-18) [3e35b05] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208). >2014-08-11 16:09:23,500 ERROR [org.ovirt.engine.core.utils.timer.SchedulerUtilQuartzImpl] (DefaultQuartzScheduler_Worker-18) Failed to invoke scheduled method _updatingTimer_Elapsed: java.lang.reflect.InvocationTargetException > at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source) [:1.7.0_55] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.lang.NoSuchMethodError: org.ovirt.vdsm.jsonrpc.client.utils.retry.RetryPolicy.<init>(IIILjava/lang/Class;)V > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.getJsonClient(JsonRpcUtils.java:48) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.createClient(JsonRpcUtils.java:38) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.createStompClient(JsonRpcUtils.java:22) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.TransportFactory.createIrsServer(TransportFactory.java:21) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData.getIrsProxy(IrsProxyData.java:579) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand.getIrsProxy(IrsBrokerCommand.java:146) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand.executeVDSCommand(IrsBrokerCommand.java:154) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:407) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData.proceedStoragePoolStats(IrsProxyData.java:293) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData._updatingTimer_Elapsed(IrsProxyData.java:187) [vdsbroker.jar:] > ... 6 more > >2014-08-11 16:09:33,562 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-67) [2530e0a8] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-08-11 16:09:33,591 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-67) [2530e0a8] hostFromVds::selectedVds - rack, spmStatus SPM, storage pool Default >2014-08-11 16:09:33,594 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-67) [2530e0a8] Initialize Irs proxy from vds: 10.34.62.208 >2014-08-11 16:09:33,600 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-67) [2530e0a8] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208). >2014-08-11 16:09:33,602 ERROR [org.ovirt.engine.core.utils.timer.SchedulerUtilQuartzImpl] (DefaultQuartzScheduler_Worker-67) Failed to invoke scheduled method _updatingTimer_Elapsed: java.lang.reflect.InvocationTargetException > at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source) [:1.7.0_55] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.lang.NoSuchMethodError: org.ovirt.vdsm.jsonrpc.client.utils.retry.RetryPolicy.<init>(IIILjava/lang/Class;)V > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.getJsonClient(JsonRpcUtils.java:48) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.createClient(JsonRpcUtils.java:38) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.createStompClient(JsonRpcUtils.java:22) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.TransportFactory.createIrsServer(TransportFactory.java:21) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData.getIrsProxy(IrsProxyData.java:579) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand.getIrsProxy(IrsBrokerCommand.java:146) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand.executeVDSCommand(IrsBrokerCommand.java:154) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:407) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData.proceedStoragePoolStats(IrsProxyData.java:293) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData._updatingTimer_Elapsed(IrsProxyData.java:187) [vdsbroker.jar:] > ... 6 more > >2014-08-11 16:09:43,656 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-68) [37aaaaff] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-08-11 16:09:43,722 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-68) [37aaaaff] hostFromVds::selectedVds - rack, spmStatus SPM, storage pool Default >2014-08-11 16:09:43,726 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-68) [37aaaaff] Initialize Irs proxy from vds: 10.34.62.208 >2014-08-11 16:09:43,732 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-68) [37aaaaff] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208). >2014-08-11 16:09:43,733 ERROR [org.ovirt.engine.core.utils.timer.SchedulerUtilQuartzImpl] (DefaultQuartzScheduler_Worker-68) Failed to invoke scheduled method _updatingTimer_Elapsed: java.lang.reflect.InvocationTargetException > at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source) [:1.7.0_55] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.lang.NoSuchMethodError: org.ovirt.vdsm.jsonrpc.client.utils.retry.RetryPolicy.<init>(IIILjava/lang/Class;)V > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.getJsonClient(JsonRpcUtils.java:48) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.createClient(JsonRpcUtils.java:38) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.createStompClient(JsonRpcUtils.java:22) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.TransportFactory.createIrsServer(TransportFactory.java:21) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData.getIrsProxy(IrsProxyData.java:579) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand.getIrsProxy(IrsBrokerCommand.java:146) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand.executeVDSCommand(IrsBrokerCommand.java:154) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:407) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData.proceedStoragePoolStats(IrsProxyData.java:293) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData._updatingTimer_Elapsed(IrsProxyData.java:187) [vdsbroker.jar:] > ... 6 more > >2014-08-11 16:09:53,786 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-97) [3c896365] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-08-11 16:09:53,849 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-97) [3c896365] hostFromVds::selectedVds - rack, spmStatus SPM, storage pool Default >2014-08-11 16:09:53,853 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-97) [3c896365] Initialize Irs proxy from vds: 10.34.62.208 >2014-08-11 16:09:53,858 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-97) [3c896365] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208). >2014-08-11 16:09:53,860 ERROR [org.ovirt.engine.core.utils.timer.SchedulerUtilQuartzImpl] (DefaultQuartzScheduler_Worker-97) Failed to invoke scheduled method _updatingTimer_Elapsed: java.lang.reflect.InvocationTargetException > at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source) [:1.7.0_55] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.lang.NoSuchMethodError: org.ovirt.vdsm.jsonrpc.client.utils.retry.RetryPolicy.<init>(IIILjava/lang/Class;)V > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.getJsonClient(JsonRpcUtils.java:48) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.createClient(JsonRpcUtils.java:38) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.createStompClient(JsonRpcUtils.java:22) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.TransportFactory.createIrsServer(TransportFactory.java:21) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData.getIrsProxy(IrsProxyData.java:579) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand.getIrsProxy(IrsBrokerCommand.java:146) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand.executeVDSCommand(IrsBrokerCommand.java:154) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:407) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData.proceedStoragePoolStats(IrsProxyData.java:293) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData._updatingTimer_Elapsed(IrsProxyData.java:187) [vdsbroker.jar:] > ... 6 more > >2014-08-11 16:10:03,916 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-21) [2c1a1597] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-08-11 16:10:03,947 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-21) [2c1a1597] hostFromVds::selectedVds - rack, spmStatus SPM, storage pool Default >2014-08-11 16:10:03,950 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-21) [2c1a1597] Initialize Irs proxy from vds: 10.34.62.208 >2014-08-11 16:10:03,956 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-21) [2c1a1597] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208). >2014-08-11 16:10:03,958 ERROR [org.ovirt.engine.core.utils.timer.SchedulerUtilQuartzImpl] (DefaultQuartzScheduler_Worker-21) Failed to invoke scheduled method _updatingTimer_Elapsed: java.lang.reflect.InvocationTargetException > at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source) [:1.7.0_55] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.lang.NoSuchMethodError: org.ovirt.vdsm.jsonrpc.client.utils.retry.RetryPolicy.<init>(IIILjava/lang/Class;)V > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.getJsonClient(JsonRpcUtils.java:48) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.createClient(JsonRpcUtils.java:38) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.createStompClient(JsonRpcUtils.java:22) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.TransportFactory.createIrsServer(TransportFactory.java:21) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData.getIrsProxy(IrsProxyData.java:579) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand.getIrsProxy(IrsBrokerCommand.java:146) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand.executeVDSCommand(IrsBrokerCommand.java:154) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:407) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData.proceedStoragePoolStats(IrsProxyData.java:293) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData._updatingTimer_Elapsed(IrsProxyData.java:187) [vdsbroker.jar:] > ... 6 more > >2014-08-11 16:10:14,013 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-48) [27d34e72] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-08-11 16:10:14,078 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-48) [27d34e72] hostFromVds::selectedVds - rack, spmStatus SPM, storage pool Default >2014-08-11 16:10:14,081 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-48) [27d34e72] Initialize Irs proxy from vds: 10.34.62.208 >2014-08-11 16:10:14,088 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-48) [27d34e72] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208). >2014-08-11 16:10:14,089 ERROR [org.ovirt.engine.core.utils.timer.SchedulerUtilQuartzImpl] (DefaultQuartzScheduler_Worker-48) Failed to invoke scheduled method _updatingTimer_Elapsed: java.lang.reflect.InvocationTargetException > at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source) [:1.7.0_55] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.lang.NoSuchMethodError: org.ovirt.vdsm.jsonrpc.client.utils.retry.RetryPolicy.<init>(IIILjava/lang/Class;)V > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.getJsonClient(JsonRpcUtils.java:48) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.createClient(JsonRpcUtils.java:38) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.createStompClient(JsonRpcUtils.java:22) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.TransportFactory.createIrsServer(TransportFactory.java:21) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData.getIrsProxy(IrsProxyData.java:579) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand.getIrsProxy(IrsBrokerCommand.java:146) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand.executeVDSCommand(IrsBrokerCommand.java:154) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:407) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData.proceedStoragePoolStats(IrsProxyData.java:293) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData._updatingTimer_Elapsed(IrsProxyData.java:187) [vdsbroker.jar:] > ... 6 more > >2014-08-11 16:10:24,162 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-23) [3ea4a3b8] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-08-11 16:10:24,183 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-23) [3ea4a3b8] Correlation ID: 3ea4a3b8, Call Stack: null, Custom Event ID: -1, Message: Data Center is being initialized, please wait for initialization to complete. >2014-08-11 16:10:24,210 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-23) [3ea4a3b8] hostFromVds::selectedVds - rack, spmStatus SPM, storage pool Default >2014-08-11 16:10:24,217 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-23) [3ea4a3b8] Initialize Irs proxy from vds: 10.34.62.208 >2014-08-11 16:10:24,227 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-23) [3ea4a3b8] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208). >2014-08-11 16:10:24,234 ERROR [org.ovirt.engine.core.utils.timer.SchedulerUtilQuartzImpl] (DefaultQuartzScheduler_Worker-23) Failed to invoke scheduled method _updatingTimer_Elapsed: java.lang.reflect.InvocationTargetException > at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source) [:1.7.0_55] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.lang.NoSuchMethodError: org.ovirt.vdsm.jsonrpc.client.utils.retry.RetryPolicy.<init>(IIILjava/lang/Class;)V > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.getJsonClient(JsonRpcUtils.java:48) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.createClient(JsonRpcUtils.java:38) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.createStompClient(JsonRpcUtils.java:22) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.TransportFactory.createIrsServer(TransportFactory.java:21) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData.getIrsProxy(IrsProxyData.java:579) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand.getIrsProxy(IrsBrokerCommand.java:146) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand.executeVDSCommand(IrsBrokerCommand.java:154) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:407) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData.proceedStoragePoolStats(IrsProxyData.java:293) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData._updatingTimer_Elapsed(IrsProxyData.java:187) [vdsbroker.jar:] > ... 6 more > >2014-08-11 16:10:34,340 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-62) [170dbdb3] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-08-11 16:10:34,405 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-62) [170dbdb3] hostFromVds::selectedVds - rack, spmStatus SPM, storage pool Default >2014-08-11 16:10:34,411 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-62) [170dbdb3] Initialize Irs proxy from vds: 10.34.62.208 >2014-08-11 16:10:34,418 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-62) [170dbdb3] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208). >2014-08-11 16:10:34,420 ERROR [org.ovirt.engine.core.utils.timer.SchedulerUtilQuartzImpl] (DefaultQuartzScheduler_Worker-62) Failed to invoke scheduled method _updatingTimer_Elapsed: java.lang.reflect.InvocationTargetException > at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source) [:1.7.0_55] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.lang.NoSuchMethodError: org.ovirt.vdsm.jsonrpc.client.utils.retry.RetryPolicy.<init>(IIILjava/lang/Class;)V > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.getJsonClient(JsonRpcUtils.java:48) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.createClient(JsonRpcUtils.java:38) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.createStompClient(JsonRpcUtils.java:22) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.TransportFactory.createIrsServer(TransportFactory.java:21) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData.getIrsProxy(IrsProxyData.java:579) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand.getIrsProxy(IrsBrokerCommand.java:146) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand.executeVDSCommand(IrsBrokerCommand.java:154) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:407) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData.proceedStoragePoolStats(IrsProxyData.java:293) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData._updatingTimer_Elapsed(IrsProxyData.java:187) [vdsbroker.jar:] > ... 6 more > >2014-08-11 16:10:44,511 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-56) [3e89abaf] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-08-11 16:10:44,581 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-56) [3e89abaf] hostFromVds::selectedVds - rack, spmStatus SPM, storage pool Default >2014-08-11 16:10:44,588 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-56) [3e89abaf] Initialize Irs proxy from vds: 10.34.62.208 >2014-08-11 16:10:44,598 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-56) [3e89abaf] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208). >2014-08-11 16:10:44,604 ERROR [org.ovirt.engine.core.utils.timer.SchedulerUtilQuartzImpl] (DefaultQuartzScheduler_Worker-56) Failed to invoke scheduled method _updatingTimer_Elapsed: java.lang.reflect.InvocationTargetException > at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source) [:1.7.0_55] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.lang.NoSuchMethodError: org.ovirt.vdsm.jsonrpc.client.utils.retry.RetryPolicy.<init>(IIILjava/lang/Class;)V > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.getJsonClient(JsonRpcUtils.java:48) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.createClient(JsonRpcUtils.java:38) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.createStompClient(JsonRpcUtils.java:22) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.TransportFactory.createIrsServer(TransportFactory.java:21) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData.getIrsProxy(IrsProxyData.java:579) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand.getIrsProxy(IrsBrokerCommand.java:146) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand.executeVDSCommand(IrsBrokerCommand.java:154) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:407) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData.proceedStoragePoolStats(IrsProxyData.java:293) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData._updatingTimer_Elapsed(IrsProxyData.java:187) [vdsbroker.jar:] > ... 6 more > >2014-08-11 16:10:54,775 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-80) [7c187b0c] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-08-11 16:10:54,865 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-80) [7c187b0c] hostFromVds::selectedVds - rack, spmStatus SPM, storage pool Default >2014-08-11 16:10:54,869 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-80) [7c187b0c] Initialize Irs proxy from vds: 10.34.62.208 >2014-08-11 16:10:54,876 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-80) [7c187b0c] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208). >2014-08-11 16:10:54,877 ERROR [org.ovirt.engine.core.utils.timer.SchedulerUtilQuartzImpl] (DefaultQuartzScheduler_Worker-80) Failed to invoke scheduled method _updatingTimer_Elapsed: java.lang.reflect.InvocationTargetException > at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source) [:1.7.0_55] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.lang.NoSuchMethodError: org.ovirt.vdsm.jsonrpc.client.utils.retry.RetryPolicy.<init>(IIILjava/lang/Class;)V > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.getJsonClient(JsonRpcUtils.java:48) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.createClient(JsonRpcUtils.java:38) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.createStompClient(JsonRpcUtils.java:22) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.TransportFactory.createIrsServer(TransportFactory.java:21) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData.getIrsProxy(IrsProxyData.java:579) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand.getIrsProxy(IrsBrokerCommand.java:146) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand.executeVDSCommand(IrsBrokerCommand.java:154) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:407) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData.proceedStoragePoolStats(IrsProxyData.java:293) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData._updatingTimer_Elapsed(IrsProxyData.java:187) [vdsbroker.jar:] > ... 6 more > >2014-08-11 16:11:04,959 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-45) [76e7601e] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-08-11 16:11:05,030 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-45) [76e7601e] hostFromVds::selectedVds - rack, spmStatus SPM, storage pool Default >2014-08-11 16:11:05,034 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-45) [76e7601e] Initialize Irs proxy from vds: 10.34.62.208 >2014-08-11 16:11:05,045 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-45) [76e7601e] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208). >2014-08-11 16:11:05,046 ERROR [org.ovirt.engine.core.utils.timer.SchedulerUtilQuartzImpl] (DefaultQuartzScheduler_Worker-45) Failed to invoke scheduled method _updatingTimer_Elapsed: java.lang.reflect.InvocationTargetException > at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source) [:1.7.0_55] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.lang.NoSuchMethodError: org.ovirt.vdsm.jsonrpc.client.utils.retry.RetryPolicy.<init>(IIILjava/lang/Class;)V > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.getJsonClient(JsonRpcUtils.java:48) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.createClient(JsonRpcUtils.java:38) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.createStompClient(JsonRpcUtils.java:22) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.TransportFactory.createIrsServer(TransportFactory.java:21) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData.getIrsProxy(IrsProxyData.java:579) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand.getIrsProxy(IrsBrokerCommand.java:146) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand.executeVDSCommand(IrsBrokerCommand.java:154) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:407) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData.proceedStoragePoolStats(IrsProxyData.java:293) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData._updatingTimer_Elapsed(IrsProxyData.java:187) [vdsbroker.jar:] > ... 6 more > >2014-08-11 16:11:15,099 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-43) [4b23bf35] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-08-11 16:11:15,169 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-43) [4b23bf35] hostFromVds::selectedVds - rack, spmStatus SPM, storage pool Default >2014-08-11 16:11:15,177 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-43) [4b23bf35] Initialize Irs proxy from vds: 10.34.62.208 >2014-08-11 16:11:15,184 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-43) [4b23bf35] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208). >2014-08-11 16:11:15,185 ERROR [org.ovirt.engine.core.utils.timer.SchedulerUtilQuartzImpl] (DefaultQuartzScheduler_Worker-43) Failed to invoke scheduled method _updatingTimer_Elapsed: java.lang.reflect.InvocationTargetException > at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source) [:1.7.0_55] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.lang.NoSuchMethodError: org.ovirt.vdsm.jsonrpc.client.utils.retry.RetryPolicy.<init>(IIILjava/lang/Class;)V > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.getJsonClient(JsonRpcUtils.java:48) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.createClient(JsonRpcUtils.java:38) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.createStompClient(JsonRpcUtils.java:22) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.TransportFactory.createIrsServer(TransportFactory.java:21) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData.getIrsProxy(IrsProxyData.java:579) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand.getIrsProxy(IrsBrokerCommand.java:146) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand.executeVDSCommand(IrsBrokerCommand.java:154) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:407) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData.proceedStoragePoolStats(IrsProxyData.java:293) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData._updatingTimer_Elapsed(IrsProxyData.java:187) [vdsbroker.jar:] > ... 6 more > >2014-08-11 16:11:25,254 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-96) [179dd399] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-08-11 16:11:25,265 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-96) [179dd399] Correlation ID: 179dd399, Call Stack: null, Custom Event ID: -1, Message: Data Center is being initialized, please wait for initialization to complete. >2014-08-11 16:11:25,283 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-96) [179dd399] hostFromVds::selectedVds - rack, spmStatus SPM, storage pool Default >2014-08-11 16:11:25,287 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-96) [179dd399] Initialize Irs proxy from vds: 10.34.62.208 >2014-08-11 16:11:25,293 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-96) [179dd399] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208). >2014-08-11 16:11:25,295 ERROR [org.ovirt.engine.core.utils.timer.SchedulerUtilQuartzImpl] (DefaultQuartzScheduler_Worker-96) Failed to invoke scheduled method _updatingTimer_Elapsed: java.lang.reflect.InvocationTargetException > at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source) [:1.7.0_55] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.lang.NoSuchMethodError: org.ovirt.vdsm.jsonrpc.client.utils.retry.RetryPolicy.<init>(IIILjava/lang/Class;)V > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.getJsonClient(JsonRpcUtils.java:48) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.createClient(JsonRpcUtils.java:38) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.createStompClient(JsonRpcUtils.java:22) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.TransportFactory.createIrsServer(TransportFactory.java:21) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData.getIrsProxy(IrsProxyData.java:579) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand.getIrsProxy(IrsBrokerCommand.java:146) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand.executeVDSCommand(IrsBrokerCommand.java:154) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:407) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData.proceedStoragePoolStats(IrsProxyData.java:293) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData._updatingTimer_Elapsed(IrsProxyData.java:187) [vdsbroker.jar:] > ... 6 more > >2014-08-11 16:11:35,348 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-31) [76b61321] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-0000000001d9 Type: StoragePool >2014-08-11 16:11:35,382 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-31) [76b61321] hostFromVds::selectedVds - rack, spmStatus SPM, storage pool Default >2014-08-11 16:11:35,386 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-31) [76b61321] Initialize Irs proxy from vds: 10.34.62.208 >2014-08-11 16:11:35,394 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-31) [76b61321] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host rack (Address: 10.34.62.208). >2014-08-11 16:11:35,396 ERROR [org.ovirt.engine.core.utils.timer.SchedulerUtilQuartzImpl] (DefaultQuartzScheduler_Worker-31) Failed to invoke scheduled method _updatingTimer_Elapsed: java.lang.reflect.InvocationTargetException > at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source) [:1.7.0_55] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.lang.NoSuchMethodError: org.ovirt.vdsm.jsonrpc.client.utils.retry.RetryPolicy.<init>(IIILjava/lang/Class;)V > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.getJsonClient(JsonRpcUtils.java:48) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.createClient(JsonRpcUtils.java:38) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcUtils.createStompClient(JsonRpcUtils.java:22) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.TransportFactory.createIrsServer(TransportFactory.java:21) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData.getIrsProxy(IrsProxyData.java:579) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand.getIrsProxy(IrsBrokerCommand.java:146) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand.executeVDSCommand(IrsBrokerCommand.java:154) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:407) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData.proceedStoragePoolStats(IrsProxyData.java:293) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData._updatingTimer_Elapsed(IrsProxyData.java:187) [vdsbroker.jar:] > ... 6 more >
You cannot view the attachment while viewing its details because your browser does not support IFRAMEs.
View the attachment on a separate page
.
View Attachment As Raw
Actions:
View
Attachments on
bug 1128778
: 925779